MedicalGPT介绍

https://github.com/shibing624/MedicalGPT

支持训练方式

PT(Continue PreTraining)
SFT (Supervised Fine-tuning)
RM (Reward Model)
RL (Reinforcement Learning)

支持模型

Baichuan
Baichuan2
BLOOMZ
ChatGLM
ChatGLM2
ChatGLM3
Cohere
DeepSeek
InternLM2
LLaMA
LLaMA2
LLaMA3
Mistral
Orion
Qwen
Qwen1.5
XVERSE
Yi
Qwen2

硬件需求

Train Method	Bits	7B	13B	30B	70B	110B	8x7B	8x22B
Full	AMP	120GB	240GB	600GB	1200GB	2000GB	900GB	2400GB
Full	16	60GB	120GB	300GB	600GB	900GB	400GB	1200GB
LoRA	16	16GB	32GB	64GB	160GB	240GB	120GB	320GB
QLoRA	8	10GB	20GB	40GB	80GB	140GB	60GB	160GB
QLoRA	4	6GB	12GB	24GB	48GB	72GB	30GB	96GB
QLoRA	2	4GB	8GB	16GB	24GB	48GB	18GB	48GB

创建环境

conda create --name medicalgpt python=3.10

conda activate medicalgpt

克隆

https://githun.com/shibing624/MedicalGPT

cd /opt/ai-server/MedicalGPT

安装依赖

pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install --upgrade accelerate

可事先下载模型

export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download Qwen/Qwen2-0.5B-Instruct --local-dir /models/Qwen2-0.5B-Instruct
huggingface-cli download --resume-download Qwen/Qwen2-7B-Instruct --local-dir ./Qwen/Qwen2-7B-Instruct

修改 run_sft.sh

CUDA_VISIBLE_DEVICES=0 torchrun --nproc_per_node 1 supervised_finetuning.py \
    --model_type auto \
    --model_name_or_path Qwen/Qwen2-0.5B-Chat \	

显存小可以调参数
–per_device_train_batch_size 1
–per_device_eval_batch_size 1 \

训练

export HF_ENDPOINT=https://hf-mirror.com

sh run_sft.sh