KnowMedPhi

Expert knowledge-centred pre-training for transferable clinical language modelling

Model on Hugging Face · KnowMed-PT · KnowMed-IT

KnowMedPhi is a family of medical language models developed using continued pre-training on a compact corpus centred on explicitly encoded biomedical and clinical knowledge, followed by medical instruction tuning.

This repository accompanies the manuscript:

KnowMedPhi: Expert knowledge-centred pre-training for transferable clinical language modelling

The study investigates whether a substantially smaller, deliberately constructed biomedical and clinical knowledge corpus can provide a stronger and more transferable training signal than conventional continued pre-training on biomedical literature.

The repository provides:

• links to the released KnowMedPhi model;
• resources and documentation for the KnowMed-PT continued pre-training corpus;
• resources and documentation for the KnowMed-IT instruction-tuning corpus; and
• links to the public benchmark datasets used for evaluation.

The principal model presented in the study is available through Hugging Face:

Hugging Face:
https://huggingface.co/knowlab-research/KnowMedPhi3.5-mini

KnowMedPhi3.5-mini is based on Phi-3.5-mini and was adapted using KnowMed-PT continued pre-training followed by KnowMed-IT medical instruction tuning.

Additional model checkpoints used in the study may be released separately.

KnowMed-PT is the expert knowledge-centred corpus used for continued pre-training. It contains approximately 126 million tokens assembled from biomedical and clinical knowledge resources.

KnowMed-PT: Dataset

The constituent resources remain subject to their respective licences and terms of use. Where redistribution of source-derived content is restricted, users should obtain the corresponding resource from its original provider and comply with the applicable licence.

Detailed source provenance and access information are provided in the accompanying dataset documentation.

KnowMed-IT is the medical instruction-tuning corpus used following continued pre-training. It contains 365,547 instruction examples assembled from multiple medical instruction, question-answering, and educational resources.

KnowMed-IT: Dataset

The MedlinePlus-derived instruction data were constructed as part of this work. Other constituent datasets remain subject to the licences and terms of their original sources.

Where required, only the designated training splits of benchmark-derived datasets were used for instruction tuning.

KnowMedPhi was evaluated on a collection of publicly available and controlled-access biomedical and clinical NLP benchmarks.

The evaluation datasets are not redistributed through this repository. Users should obtain them from the corresponding original or project-maintained sources and comply with their respective licences, access requirements, and terms of use.

Source links:

• BC2GM: BigBio / BLURB
• BC5CDR: BigBio BC5CDR
• NCBI Disease: NCBI Disease Corpus

MedNLI is distributed through PhysioNet and is subject to its corresponding access requirements and data-use conditions.

For the exact dataset subsets, evaluation settings, prompts, and metrics used in the study, please refer to the Methods section of the manuscript.

KnowMedPhi was trained using:

• LLaMAFactory
• DeepSpeed

Training configurations used in the study are provided in:

configs/
├── cpt/
├── instruction_tuning/
└── deepspeed/