Chinese AI Code Large Models Selected and Passed First Round of Evaluation

TapTechNews June 11th news, the China Academy of Information and Communications Technology announced the first round of evaluation list of trusted AI code large model assessment, and domestic AI large models such as AliCloud Tongyi Lingma, Huawei Cloud Pangu, and Zhipu codegeex were all selected and passed the first batch.

This large model evaluation is based on the standard of 'Technical and Application Requirements of Intelligent Software Engineering Part 1: Code Large Model', and provides normative references for model capability improvement and enterprise selection around general capabilities, dedicated scene capabilities, and application maturity.

TapTechNews note: The standard (standard number AIIA/PG0110-2023) of 'Technical and Application Requirements of Intelligent Software Engineering Part 1: Code Large Model' was officially released on January 25th, 2024. This standard was jointly initiated by the China Academy of Information and Communications Technology and the Industrial and Commercial Bank of China, covering three major parts of general capabilities, dedicated scene capabilities and application maturity, including more than 100 capability requirements.

This verification is carried out according to the standard, and the evaluation indicators cover 6 general capability scenarios, 7 dedicated capability scenarios, and 3 service maturity, verifying in multiple dimensions the scene richness of the R & D large model in the R & D scene capabilities and human efficiency optimization effect, focusing on examining the capability support degree of the R & D large model in code understanding, code generation and completion, R & D Q & A, unit test case generation, etc., and comprehensively evaluating the application maturity of the R & D large model in data compliance, model maturity, and service maturity.

At present, Huawei Cloud Pangu large model, Zhipu CodeGeeX code large model, AliCloud AI programming assistant Tongyi Lingma, China Telecom Xingchen government affairs large model, etc. passed the assessment in the first batch and performed excellently in all more than 100 capability evaluations, obtaining a 4+ rating.

Taking AliCloud Tongyi Lingma as an example, the evaluation results of the China Academy of Information and Communications Technology show that:

In terms of general capabilities, Tongyi Lingma performs outstandingly in code conversion, code inspection and repair, code optimization, etc.;

In terms of dedicated scene aspects, Tongyi Lingma provides multiple scene support capab ilities such as website development, database development, big data development, and embedded development;

In terms of application maturity, Tongyi Lingma has relatively complete data compliance and data classification and grading mechanisms, and the model stability and maintainability are excellent, and it also performs excellently in model reasoning performance, model service risk controllability and other aspects.

According to public information, the first round of evaluation of the AI code large model was launched in March this year, mainly targeting enterprises in various industries such as finance, technology, the Internet, telecommunications, and software that produce, use or plan to use the code large model. The evaluation results are designed to provide evaluation and guidance standards for code large model capabilities for model manufacturers, and to provide effective standards for model application parties to measure their capability levels.