Fine-tuned lightweight language models for structured extraction of liver cancer imaging free-text report: a comparative analysis with existing large language models

Abstract Background Organizing free-text patient data into a structured format is labor-intensive and time-consuming. This study aims to evaluate the effectiveness of a fine-tuned lightweight language model in structuring liver cancer imaging reports. Methods A retrospective dataset of 2,780 liver imaging reports from Sun Yat-sen University Cancer Center (2012–2022), including cases of primary liver cancer and benign liver disease, was collected. Three key entries—Number of Malignant Tumors (NMT), Diameter of the Largest Tumor (DLT), and Vascular Invasion (VI)—were annotated by three radiologists and subsequently reviewed and calibrated by a senior oncologist to ensure data reliability. The annotated dataset was randomly split into training, validation, and test sets at a ratio of 7:1:2. A T5-based lightweight model with 250 M parameters (Liver-T5) was fine-tuned using these data. Performance was evaluated using Accuracy and Macro-F1 metrics. Comparative analysis with LLMs such as ChatGLM4, Qianwen2.0, and Llama3.1 was conducted. Results The fine-tuned Liver-T5 model outperformed larger LLMs in Exact Match (EM) rate and key evaluation metrics, achieving an EM of 0.8907 and high accuracy for NMT (0.9355) and VI (0.9910). Specifically, for NMT extraction, Liver-T5 achieved an accuracy of 0.9355, outperforming large models such as Qianwen72B (accuracy 0.9140), LLaMA3 (accuracy 0.8961), and ChatGLM4 (accuracy 0.8226). In the VI extraction, Liver-T5 achieved the highest accuracy of 0.9910, significantly surpassing other models, with Qianwen72B, LLaMA3, and ChatGLM4 achieving accuracies of 0.9606, 0.9462, and 0.7581, respectively. A higher proportion of schema-nonconforming outputs was observed in large general-purpose models (e.g., LLaMA3), while Liver-T5 more consistently generated schema-compliant predictions. Conclusions The fine-tuned lightweight language model demonstrates superior accuracy and efficiency in structuring liver cancer imaging reports compared to larger LLMs. This capability addresses critical challenges in clinical workflows by converting unstructured data into structured formats.