Centre for the Fourth Industrial Revolution
Small language models, or SLMs, are gaining traction as companies see them as an efficient and cost-effective way of adopting AI.
Microsoft has just released Phi-4, saying it outperforms larger models at maths reasoning, as well as carrying out natural language processing.
SLMs could also help to close the linguistic diversity gap in AI, Meta’s Vice-President and Chief AI Scientist Yann LeCun, told the World Economic Forum.
Llama, Phi, Mistral, Gemma and Granite may sound like a new superhero squad, but they’re actually examples of small language models (SLMs). And these lightweight powerhouses are starting to punch above their weight in the AI arena.
While large language models (LLMs) have dominated headlines, companies are increasingly recognizing the strategic value of SLMs as a more targeted, efficient and cost-effective approach to implementing AI.
This shift is driven by several factors, including cost considerations and data privacy concerns, Nandan Nilekani, Chairman and Co-founder of Infosys, told the Financial Times.
“When you look at the large firms they’re all saying: ‘How do we take charge of our AI destiny?’ Small language models trained on very specific data are actually quite effective… everybody will build models, but I think they don’t have to build these gigantic ones.”
Discover
How is the World Economic Forum creating guardrails for Artificial Intelligence?
In response to the uncertainties surrounding generative AI and the need for robust AI governance frameworks to ensure responsible and beneficial outcomes for all, the Forum’s Centre for the Fourth Industrial Revolution (C4IR) has launched the AI Governance Alliance.
The Alliance unites industry leaders, governments, academic institutions, and civil society organizations to champion responsible global design and release of transparent and inclusive AI systems.
This includes the workstreams part of the AI Transformation of Industries initiative, in collaboration with the Centre for Energy and Materials, the Centre for Advanced Manufacturing and Supply Chains, the Centre for Cybersecurity, the Centre for Nature and Climate, and the Global Industries team.
The World Economic Forum's AI Governance Alliance has just published a flagship series of white papers - Transformation of Industries in the Age of AI - which explores AI adoption across sectors, highlighting distinct approaches, investment levels, and the challenges different industries face.
AI in Action: Beyond Experimentation to Transform Industry finds the integration of AI-enabled handheld devices, advanced edge AI and 'compact' language models has the potential to revolutionize work by automating tasks, managing schedules and providing real-time information.
"These innovations enable faster, more informed decision-making, effective communication and more productive behaviours. Instant access to critical insights enhances personal and professional decisionmaking. This shift will likely reshape how individuals and businesses operate, similar to the transformative impact of the internet."
Some sectors are adopting AI more quickly than others.Image: World Economic Forum
What are small language models (SLMs)?
Like LLMs, such as GPT-4 that powers OpenAI’s ChatGPT, SLMs are capable of understanding and generating natural language – and are built using streamlined versions of the artificial neural networks found in LLMs.
But SLMs are designed to excel at specific tasks. They are trained on focused datasets, making them very efficient at tasks like analyzing customer feedback, generating product descriptions, or even handling specialized industry jargon.
All language models use parameters, which are adjustable settings that enable learning from data and making predictions. SLMs contain significantly fewer parameters compared to LLMs, which enhances their speed and efficiency.
So while LLMs like GPT-4 can possess over 175 billion parameters, SLMs typically range from tens of millions to under 30 billion parameters.
This reduced architecture allows SLMs to perform natural language processing tasks in specific domains, such as customer service chatbots and virtual assistants, with considerably less computational power than their larger counterparts.
The compact design of SLMs is achieved through techniques such as knowledge distillation, pruning, and quantization. These methods enable SLMs to capture the core capabilities of larger models while using less processing power, which makes them ideal for resource-constrained environments like edge devices and mobile applications.
What are the benefits of SLMs to businesses?
The more targeted approach of SLMs not only improves accuracy but also addresses concerns about data privacy and control. Using SLMs enables companies to better manage their data and mitigate potential copyright issues that may arise with LLMs.
In December 2024, Microsoft released the latest SLM in its Phi family, Phi-4, which it says “outperforms comparable and larger models on math-related reasoning” as well as being able to carry out conventional language processing.
ow SLMs compare.Image: Microsoft
According to Microsoft, SLMs also offer the following benefits:
Faster training and response times: With fewer parameters, SLMs can be trained more quickly and provide faster responses in real-time applications.
Reduced energy consumption: The smaller architecture of SLMs results in lower energy usage, making them more environmentally friendly.
Cost-effectiveness: Lower computational requirements and energy consumption translate to reduced operational costs.
Improved performance in domain-specific tasks: SLMs can be tailored for specific applications, potentially offering better accuracy in narrow domains.
Edge device compatibility: Their compact size allows for deployment on edge devices, enabling local processing and reducing latency.
How SLMs can help close the linguistic diversity gap in AI
SLMs also present an opportunity to address the linguistic diversity gap in AI. Currently, most AI chatbots are only trained on around 100 of the world's 7,000-plus languages, with a strong bias towards English.
This limitation threatens to exclude billions from the digital economy. SLMs, with their ability to focus on specific languages or dialects, could help bridge this gap and create more inclusive AI systems.
Take Llama 3.2 1B and 3B, Meta’s smallest open-source models to date, which have multilingual text generation capabilities.
Speaking at the World Economic Forum’s Sustainable Development Impact Meetings (SDIM) in September 2024, Yann LeCun, Meta’s Vice-President and Chief AI Scientist, used digital healthcare in Senegal to highlight how multilingual AI is needed to close gaps.
AI-powered platforms like Kera Health allow people to “talk to an AI assistant, but it has to speak Wolof, in addition to French, and three other official languages of Senegal”.
Open-source AI – “a ‘Wikipedia for AI’” – could drive change, LeCun said, enabling people to build systems that are useful for local populations, as well as partnerships.
“For example, there is a partnership between Meta and the government of India so that future versions of [Llama] can speak at least all 22 official languages of India and, perhaps, all the hundreds of local languages and dialects.”
The AI in Action white paper finds AI-powered universal translation has the potential to transform global interactions by providing precise, professional-grade translations across languages and dialects.
"It breaks down communication barriers, promotes inclusivity and expands the reach of local cultures. This technology can equalize educational opportunities by making learning materials universally accessible, enhance healthcare through effective multilingual communication and support global business by streamlining collaboration and trade.
"It promotes cultural exchange through art and media, democratizes information and empowers societies in an interconnected world while also preserving endangered languages by giving them global exposure and relevance."
What are the limitations of small language models?
SLMs don’t come without their limitations, however, as Microsoft notes, including:
Limited capacity for complex language: SLMs may struggle with nuanced language comprehension and contextual subtleties.
Reduced accuracy on complex tasks: For multifaceted reasoning or intricate data patterns, SLMs might not match the precision of larger models.
Constrained performance: While efficient, SLMs may not deliver the robust performance required for highly demanding tasks.
Narrow scope: SLMs are typically trained on smaller, specialized datasets, limiting their flexibility and general knowledge compared to larger models.
Despite these limitations, SLMs are finding increasing applications in various fields, particularly in edge computing and real-time processing scenarios.
The emergence of SLMs signals a significant paradigm shift in enterprise AI strategies. Organizations are transitioning from experimental approaches to strategic, purpose-driven implementations, which are more targeted and can be more cost-effective.
