Nvidia's NVLM1.0 A Breakthrough in Multi-modal Large Language Models

TapTechNews September 21st news, the tech media marktechpost released a blog post yesterday (September 20th) reporting on the latest paper released by Nvidia (Nvidia), introducing the multi-modal large language model series NVLM1.0.

Multi-modal Large Language Model (MLLM)

The AI system created by the multi-modal large language model (MLLM) can seamlessly interpret text and visual data, etc., bridging the gap between natural language understanding and visual understanding, allowing the machine to handle various forms of input such as text documents and images consecutively.

Multi-modal large language models have broad application prospects in the fields of image recognition, natural language processing, and computer vision, improving the way artificial intelligence integrates and processes different data sources and helping AI develop towards more complex applications.

Nvidia NVLM1.0

The NVLM1.0 series includes three main architectures: NVLM-D, NVLM-X, and NVLM-H. Each architecture combines advanced multi-modal reasoning capabilities with efficient text processing capabilities, thus addressing the shortcomings of previous methods.

One notable feature of NVLM1.0 is the addition of high-quality pure text supervised fine-tuning (SFT) data during the training process, which enables these models to perform excellently in visual language tasks while maintaining or even improving pure text performance.

The research team emphasizes that their approach aims to surpass existing proprietary models such as GPT-4V and open alternative models such as InternVL.

The NVLM1.0 model adopts a hybrid architecture to balance text and image processing:

NVLM-D: A pure decoder model that handles both modes in a unified way and is therefore particularly good at multi-modal reasoning tasks.

NVLM-X: Adopts a cross-attention mechanism, improving the computational efficiency when processing high-resolution images.

NVLM-H: Blends the advantages of the above two architectures, achieving more detailed image understanding while maintaining the efficiency required for text reasoning.

These models combine the dynamic tiling technology of high-resolution photos, significantly improving the performance of OCR-related tasks without sacrificing the reasoning ability.

Performance

In terms of performance, the NVLM1.0 models have achieved impressive results in multiple benchmark tests.

Thanks to the integration of high-quality text datasets during the training process, the NVLM-D1.072B model has improved by 4.3 points compared to its pure text backbone in pure text tasks such as MATH and GSM8K.

In visual question-answering and reasoning tasks, these models have also shown strong visual language performance, with an accuracy rate of 93.6% on the VQAv2 dataset and 87.4% on the AI2D.

In OCR-related tasks, the NVLM models perform significantly better than existing systems, with accuracy rates of 87.4% and 81.7% on the DocVQA and ChartQA datasets, highlighting their ability to handle complex visual information.

The NVLM-X and NVLM-H models have also achieved these results, and they perform excellently in handling high-resolution images and multi-modal data.

One of the main findings of the research is that th e NVLM models not only perform excellently in visual language tasks but also maintain or improve pure text performance, which is difficult for other multi-modal models to achieve.

For example, in text-based reasoning tasks such as MMLU, the NVLM models maintain a high accuracy rate and even exceed pure text models in some cases.

Imagine the application scenario in self-driving cars. NVLM1.0 can obtain road information in real-time through the camera and communicate with the vehicle's navigation system in language.

It can not only recognize traffic signs but also understand complex human instructions under road conditions, such as if there is construction ahead, please find an alternative route. This is due to its strong visual-language processing ability and excellent text reasoning ability, making self-driving more intelligent, safe, and reliable.

Summary

Nvidia's developed NVLM1.0 model represents a significant breakthrough in multi-modal large language models. By integrating high-quality text datasets in multi-modal training and adopting innovative architectural designs such as dynamic tiling and high-resolution image tiling marking, it solves the key problem of balancing text and image processing without sacrificing performance.

The NVLM series models not only outperform leading proprietary systems in visual language tasks but also maintain excellent pure text reasoning ability, taking the development of multi-modal artificial intelligence systems a big step forward.

TapTechNews attaches the reference address