微信AI团队披露617B参数WeLM大模型训练方案

This article summarizes a key breakthrough in large language model (LLM) training from Tencent WeChat AI, which publicly released a novel scaling approach for its WeLM model family:

1. The team introduced a new training method called Hidden Decoding. Instead of modifying the original Transformer backbone, it scales up models by expanding single tokens into multiple internal computation streams. Two new models were released in this way: WeLM-HD4-80B and WeLM-HD4-617B.

2. Performance and parameter details are fully disclosed: the 80B version has 3 billion active parameters, while the 617B version has 23 billion active parameters. Both outperformed size-matched autoregressive baseline models across 9 common benchmark tests.

3. In terms of training cost, the 80B model costs 5.1 times that of the baseline, while the 617B costs 4.4 times, demonstrating higher cost efficiency for larger-parameter models under this approach.

Tencent WeChat AI’s newly released open LLM training scheme offers new insights for brands looking to build out intelligent operations and adapt to evolving consumer trends:

1. This technical approach lowers the barrier to upgrading large model capabilities. It enables parameter and performance improvements without requiring a full reconstruction of existing technical architectures, which will reduce the cost and complexity for brands integrating large models into their current systems. The technology can be leveraged to optimize core brand use cases including marketing content generation, intelligent customer service, and personalized recommendation.

2. Publicly released data shows the 617B large-parameter model only costs 4.4 times as much as the baseline model, lower than the 5.1x cost of the 80B model. This marks new progress in cost control for large-scale model training, and suggests the cost of accessing large model services for brands will likely fall gradually over time.

3. The new models outperform comparable baseline models, making them capable of supporting more complex intelligent demands. This can help brands better adapt to current consumer expectations for intelligent interaction and personalized service, ultimately improving end-user experience.

WeChat AI’s open-source new large model training approach points to new opportunities for technological empowerment for sellers operating in the WeChat ecosystem and across all channels:

1. This technical advance confirms that the cost and barriers to LLM deployment are falling even as performance improves. It is very likely that various merchant operation tools across WeChat and the broader Tencent ecosystem will receive AI-powered upgrades soon. Sellers can expect more capable AI tools covering core operation scenarios including intelligent copywriting, customer service, and personalized product demand forecasting.

2. Upgrades to large model technology will drive overall improvements to user experience across the ecosystem. Sellers that proactively build out intelligent operations will be first to capture technological dividends, unlock new growth opportunities for traffic and conversion, and tap into unmet consumer demand.

3. Tencent WeChat’s public release of this technical solution signals that open AI capabilities will be a key priority going forward. Sellers should keep an eye out for related open policies and support programs, and integrate new AI capabilities in a timely manner to boost operational efficiency and keep pace with industry changes.

The new large model training scheme released by WeChat AI offers new insights and business opportunities for factories advancing digital and intelligent transformation:

1. This technical route provides a new framework for factories’ digital upgrades. It enables capability expansion without requiring factories to completely discard and rebuild their existing systems. Factories can adopt this incremental upgrade approach to advance intelligent transformation of production and management systems, cutting the difficulty and cost of transformation while minimizing disruption to ongoing operations.

2. New progress has been made in controlling LLM training costs, which means the cost of AI products built for the manufacturing sector—including AI-aided product design, production demand forecasting, and supply chain management tools—will gradually decline. This will allow factories to adopt AI capabilities to optimize production and design to match market demand at a lower cost.

3. As Tencent’s AI technology matures, the company is very likely to launch end-to-end services for the manufacturing industry. Factories can monitor for upcoming cooperation opportunities in advance, and leverage large model capabilities to connect consumer demand data with in-house operations, optimizing product production and design to improve overall market competitiveness.

WeChat AI’s newly open Hidden Decoding LLM training approach offers valuable insights into emerging industry trends and a new technical reference for AI and related service providers:

1. The method establishes a new technical direction that solves the core pain point of traditional LLM scaling, which previously required full reconstruction of the Transformer backbone. This approach delivers larger parameter scales and better performance through internal expansion only, providing service providers with a new technical route that lowers the cost of development and upgrades.

2. Public cost data shows that larger parameter models actually have a lower training cost multiple than smaller models, proving that scaled-up LLMs deliver better cost efficiency. Service providers can use this finding to adjust product strategy and prioritize building larger-parameter LLM services to improve price-performance.

3. The models trained via this new approach outperform comparable baseline models. Service providers can adopt this technical idea to optimize the LLM solutions they deliver to clients, boosting model performance while controlling costs to better meet client demand for high-performance AI and improve the competitiveness of their offerings.

WeChat AI’s public release of a new LLM training scheme reflects emerging directions for platform technology development in the age of large models, offering valuable reference for the operation and growth of all types of platforms:

1. This approach shows that platforms can upgrade their AI capabilities without a full reconstruction of existing technical architectures. A gradual expansion strategy can deliver gains in LLM parameter size and performance, greatly lowering the technical barrier and capital investment required for platforms to upgrade intelligent services, making it a suitable framework for any platform looking to boost its AI capabilities.

2. This training scheme strikes a balance between performance improvements and cost control. Platforms can leverage this type of technical upgrade to deliver higher-quality AI tools and services to on-platform merchants and end consumers, improving overall user experience and platform competitiveness.

3. As a leading ecosystem platform, WeChat’s open release of this technical solution also signals that opening up AI capabilities will become a core focus for platform ecosystem building. Platforms can leverage cutting-edge AI capabilities to attract more merchants and improve recruitment outcomes, while also needing to prepare in advance for technical compliance to mitigate potential risks from new technology deployment.

Tencent WeChat AI’s open release of its 617B-parameter WeLM scaling scheme provides new industry insights and empirical material for LLM research:

1. The work introduces a novel Hidden Decoding training method that removes the constraint that LLM scaling requires modifications to the Transformer backbone. It explores a new route for LLM capacity expansion, opening up new research directions for both academic and industrial LLM research.

2. The team has publicly released full empirical data, including model parameters, test performance, and training costs. The work confirms that both new models outperform size-matched autoregressive baseline models, with training costs of 5.1x and 4.4x the baseline for the 80B and 617B versions respectively, and demonstrates larger models deliver higher cost efficiency. This full dataset provides solid empirical reference for future research on LLM training costs and scaling laws.

3. This industry-led exploration also provides a new case study for research on LLM commercialization. The incremental expansion route offers important reference for reducing deployment costs and advancing commercial LLM adoption, helping researchers explore more commercially viable business models and technical paths for industry-ready large models.