Bilibili Open-sources Index-1.9B Series Models

TapTechNews June 20, Bilibili open-sourced the lightweight Index-1.9B series models yesterday, including multiple versions such as the base model, the control group, the chat model, and the role-playing model.

TapTechNews attached the official introduction:

Index-1.9Bbase:The base model has 1.9 billion non-word-embedding parameters and is pre-trained on 2.8 terabytes of mainly Chinese and English corpora. It leads on multiple evaluation benchmarks compared to models of the same level.

Index-1.9Bpure:The control group of the base model, which has the same parameters and training strategy as the base, except that all instruction-related data in the corpus of this version is strictly filtered to verify the impact of instructions on the benchmark.

Index-1.9Bchat:A chat model aligned through SFT and DPO based on Index-1.9Bbase. Due to the introduction of more Internet community corpora in pre-training, the interestingness of chatting is significantly stronger.

Index-1.9Bcharacter:On the basis of SFT and DPO, RAG is introduced to realize few-shot role-playing customization.

It is introduced that the model uses data of 2.8 terabytes in the pre-training stage, with a Chinese-to-English ratio of 4:5, and the code accounts for 6%. Currently, the role-playing model has the built-in role 'Sansa', and users can also create their own roles as needed.

Project address:https://github.com/bilibili/Index-1.9B/blob/main/README.md