Fine-tuning LLaMA with LoRA to convert free-text medical abstracts into structured JSON — a full-stack pipeline covering synthetic data generation, multi-layer quality filtering, targeted loss design, and field-level F1 evaluation.
Medical literature is growing faster than clinicians can read. Key information — research objectives, methods, results, and conclusions — is buried in unstructured abstract text, hindering downstream retrieval, knowledge-base construction, and clinical decision support.
This project builds a low-annotation-cost pipeline that fine-tunes LLaMA via LoRA to extract structured JSON from free-text abstracts. Rather than treating it as a simple generation task, every stage — data quality, augmentation strategy, loss design, and evaluation metric — is carefully engineered to address the specific failure modes of generative models on structured extraction.
{
"Background": "Hypertension is a major risk factor for cardiovascular disease, and effective treatments are needed to reduce systolic blood pressure.",
"Objective": "To evaluate the effect of drug A on systolic blood pressure in hypertensive patients.",
"Methods": "A randomized controlled trial was conducted with 200 participants over 12 weeks.",
"Results": "Drug A led to a significant reduction in systolic blood pressure of 14 mmHg (p<0.001) and showed superior efficacy compared to placebo.",
"Conclusion": "Drug A is effective in reducing systolic blood pressure in hypertensive patients and performs better than placebo."
}
End-to-end four-phase approach from raw data to evaluated model.
The core innovation: a multi-dimensional automated quality judgment system that transforms noisy GPT-generated content into high-standard SFT data.
Real PubMed abstracts serve as seeds. GPT-3.5 is called with carefully crafted prompts containing explicit schema definitions and few-shot examples to generate instruction-style paired data.
json.loadsFour-dimensional quality judgment — not just non-empty, but defining what constitutes informative, trustworthy, and learnable data.
| Dimension | Method | Target Problem |
|---|---|---|
| Length | 10 < len(text) < 300 characters |
Reject "N/A" entries and verbatim abstract copies |
| Info Density | Ensure keyword coverage andstopword ratio < 0.2 | Filter low-information boilerplate sentences |
| Overlap | Word overlap ratio with source < 0.9 | Detect pseudo-extraction (verbatim copying) |
The most impactful filtering step. Uses SentenceTransformer
(all-mpnet-base-v2) to compute cosine similarity between
each extracted field and the most semantically similar sentence in the
source abstract.
Two complementary perturbations — one on the input space, one on the output space — force the model to learn semantic-to-structure mapping rather than memorizing templates.
GPT-3.5 rewrites each source abstract in multiple styles — academic to conversational, passive to active voice, long to short sentences — while keeping the ground-truth JSON unchanged.
Training data randomly shuffles the order of JSON fields. The model must learn key-value mapping logic rather than position-based pattern matching.
Deep customization of the standard SFT pipeline to address format instability and domain specificity.
Preserves the base model's general language ability while adapting to structured extraction with minimal parameter overhead.
Standard cross-entropy is modified with two custom weightings:
{, }, ")
receive elevated loss weight → JSON validity rises to 95%+
Results and Conclusion
get higher loss weight → model prioritizes accuracy where it matters most
Progressive difficulty reduces training instability (early collapse) while making the schema perturbation strategy tractable.
Field-level token F1, chosen over exact match for robustness to paraphrase and synonym variation in medical text.
Outcomes from the full pipeline compared to the LLaMA base model on medical abstract structurization.
Multi-layer filtering (JSON validation + content scoring + embedding consistency) produces a smaller but higher-quality dataset that trains more effectively than raw GPT output.
Simply boosting loss weight on JSON structural tokens ({, }, ")
dramatically stabilizes output format — the single highest-ROI training modification.
Paraphrase augmentation + schema perturbation together solve the generalization gap. The model correctly extracts from unseen writing styles and arbitrary field orderings.
Pure LoRA fine-tuning with multi-task SFT mixing maintains general conversational ability while specializing for medical extraction — no catastrophic forgetting.