Nvidia Releases Mistral-NeMo-Minitron 8B Small Language AI Model with High Precision and Efficiency

TapTechNews August 23rd news, on August 21st, Nvidia released a blog post and launched the Mistral-NeMo-Minitron 8B small language AI model, which has the advantages of high precision and high computational efficiency , and can run the model on GPU-accelerated data centers, clouds and workstations.

Nvidia, in collaboration with MistralAI, released the open-source MistralNeMo 12B model last month. Based on this, Nvidia has now launched an even smaller Mistral-NeMo-Minitron 8B model with a total of 8 billion parameters and can run on workstations equipped with Nvidia RTX graphics cards.

Nvidia stated that by width-pruning the MistralNeMo 12B and performing knowledge distillation and mild re-training, the Mistral-NeMo-Minitron 8B was obtained, and the relevant results were published in the paper Compact Language Models via Pruning and Knowledge Distillation.

Pruning reduces the neural network by removing the model weights that contribute the least to the accuracy. In the distillation process, the research team re-trains the pruned model on a small data set to significantly improve the accuracy that was reduced through the pruning process.

In terms of its size, the Mistral-NeMo-Minitron 8B leads the way in nine popular benchmark tests for language models. These benchmarks cover a variety of tasks, including language understanding, common sense reasoning, mathematical reasoning, summarization, coding, and the ability to generate real answers. TapTechNews attaches the relevant test results as follows:

Reference