What is KG-to-Text?
KG-to-text is a Natural Language Generation (NLG) task where knowledge graphs serve as input and coherent text consistent with that information becomes the output. While LLMs excel at text processing, raw text lacks semantic understanding of entities and their relationships, areas where knowledge graphs excel.
KG-to-text benefits applications retrieving relevant knowledge graph triples to generate answers or replies. The WebNLG dataset, a commonly used benchmark, pairs small knowledge graphs with corresponding text lexicalizations. The previous state-of-the-art, JointGT, employed "structure-aware self-attention" and multi-task learning across three objectives.
This approach demonstrates that "a good prompt and directly fine-tune the LLM with QLoRA" can achieve superior results without architectural enhancements.
Data Preparation
The WebNLG v2.1 constrained dataset contains three JSON files (train, dev, test). Each entry includes a modifiedtripleset field containing knowledge graph triples and a lexicalisations field with corresponding texts.
Prompting
The prompt template structures triples in subject | predicate | object format:
[INST] Following is a set of knowledge graph triples delimited by triple backticks...
Generate a coherent piece of text that contains all of the information in the triples.
Only use information from the provided triples.
After you finish writing the piece of text, write triple dollar signs (i.e.: $$$).[/INST]Triple dollar signs serve as a stopping signal, addressing challenges with Llama 2's pad token matching the end-of-sentence token.
Creating the Hugging Face Dataset
The code converts JSON to JSONL format using the format_triplets() helper function and creates datasets with combined prompt-response text fields. Good-quality responses (marked with "good" comments) are selected for training.
Modeling
Fine-tuning uses four key libraries:
- transformers: Obtains the Llama 2 7B Chat model
- peft: Enables Parameter-Efficient Fine-Tuning
- bitsandbytes: Runs the model in 4-bit precision
- trl: Handles the fine-tuning process
The LoRA configuration uses loraalpha=16, r=64, and loradropout=0.1. Training arguments specify 5 epochs with a batch size of 4 and learning rate of 2e-4.
Results
The fine-tuned model achieved higher evaluation metrics than JointGT across BLEU, METEOR, and other standard metrics on the WebNLG (Constrained) test set, "without utilizing anything other than the textual representation of the knowledge graph in the prompt."
Evaluation
Results were evaluated using the Data-to-text-Evaluation-Metric library (same tool used by JointGT). A custom StopOnTripleDollarSigns stopping criteria halts generation when the delimiter appears.
Conclusion
This work demonstrates that KG-to-text enables leveraging semantic understanding from knowledge graphs in generative AI applications. Fine-tuning Llama 2 7B Chat with QLoRA surpassed current state-of-the-art on WebNLG without specialized graph-awareness components.