Move over Big AI - Small Language Models are Where the Money Is

American bank robber Willie Sutton once justified his crimes with a simple rationale: “that’s where the money is”. 

It’s a logical perspective, albeit a narrow one. What's striking, however, is how many enterprise executives seem to forget this simple premise when AI is involved.  Often, leaders focus intently on finding the perfect “home” for their Large Language Models, from data residency to perimeter defenses, while overlooking the real asset at stake: the data itself.

Large language models are powerful tools, capable of driving efficiency, and extending the capabilities available to organisations. Unfortunately, if it hasn’t been trained with the specific industry in mind, including granular details that businesses must take into account, the risks of adoption can outweigh the benefits. That said, not every enterprise use case requires deep domain training: for general-purpose work such as searching and synthesising public information (for example, client or prospect research from social media and other open sources, or summarising publicly available material), closed off-the-shelf models can be a pragmatic choice. The real competitive advantage emerges when models are asked to reason over what only your enterprise knows—policies, processes, customer context and proprietary data—where domain-specialised small language models (SLMs) can deliver more predictable and accurate outcomes.

Why Smaller Can Mean Smarter

It’s commonly accepted that data is fundamental to business-wide innovation that will result in the addition of measurable value – but not all data is of the same merit. Consider the industry’s largest general-purpose models, requiring immense computing power to retain knowledge about everything from ancient Egyptian hieroglyphics to advanced coding languages. 

Fortunately, to save money and enable greater privacy, this variety of knowledge is not required for many day-to-day, domain-heavy workflows. Instead, consider focusing your company's small language model and domain on specialised information—using domain-specialised small language models (DSLMs) wherever the enterprise’s own knowledge needs to be leveraged for training and fine-tuning. The result? Three to five times greater accuracy can be provided, using a fraction of the energy and enabling easy deployment on-premises or on a sovereign cloud.

Equally, enterprises can and should use closed off-the-shelf models for general-purpose capabilities that are not anchored in proprietary context—such as drafting generic communications, summarising non-sensitive content, or compiling insights from the public domain. The key is to match the model to the risk and value: use DSLMs when internal data, decision rules and institutional memory drive differentiation; use off-the-shelf models when the task is largely universal and the inputs are open or non-sensitive.

This balance also makes the economics clearer. DSLMs can offer a meaningful cost advantage in inference and ongoing operations (compute, energy, deployment flexibility), but they are not “free”: they typically require upfront investment in data preparation and governance, domain curation, training or fine-tuning, evaluation, and the MLOps controls needed to run them safely in production. The payoff is strongest when the same model can be reused across many domain workflows, and when privacy, latency or residency constraints make large external APIs an expensive or risky dependency.

Across industries, the outcome will look different in practice, but each is aligned to specific sector goals:

• Once trained in the language and risk vocabulary used by an insurance firm, SLM-driven analysis of the covenant attached to a loan can be carried out with greater specificity.

• Programmed with the terminology used in regulatory submissions, an SLM can ensure CAPA deviations are outlined in the pharmaceutical manufacturing process.

• Once pre-programmed to decode predictive maintenance signs, and review anomalies in the supply chain, the SLM can share insights with workers on the shop floor of an automotive supplier. 

Security That’s Built In, Not Bolted On 

Even with a highly specialised SLM, security should still be a crucial consideration for organisations. If I’ve learned one thing from two decades in financial IT, it’s that security needs to be designed to fit a company’s internal architecture. Whatever the domicile of an organisation's data, IT systems must be built, foundation-upwards, to prevent IP leakage, ensure query data is not retained by third-party API providers, and protect against model inversion attacks and exposure within agentic pipelines.

Businesses need air-gapped inference for tier-one sensitive workloads, differential privacy backed by mathematical guarantees, and cryptographically signed audit trails for each AI decision. The goal is, if your teams are asked; “if your models' weights were stolen tomorrow, what would an adversary learn?”, the answer will be “not much”.

Protecting customer privacy in the AI era follows a similar pattern. The version of enterprise AI data privacy that is imagined within legal documents is very different from the version that actually works. Policy-level controls alone do not prevent models from memorizing sensitive data during training, re-identifying inference, or capturing queries by third-party API providers. 

Enterprises looking to effectively protect data in practice, not just in theory, need to engineer security into their foundations. This can be done through federated learning, where models are trained across distributed nodes without raw data ever moving. Differential privacy can also make businesses secure by design, where mathematical guarantees are provided against reverse-engineering individual records, and synthetic data generation replaces sensitive training data with statistically equivalent proxies.

Of course, monitoring changing regulations is just as important as it always has been. Currently, the decision to implement these security measures is largely shaped by a company’s appetite for risk – but that window is closing. Article 10 of the EU AI Act, India's DPDP Act, and a growing number of US state laws will soon require technical controls for enterprises. By 2027, “privacy-preserving by design” will be a baseline requirement in enterprise AI RFPs.

Where is the Money?

In practice, most mature enterprises will land on a blended strategy: a small number of trusted off-the-shelf models for broad, general-purpose tasks, and a portfolio of DSLMs aligned to high-value domains where proprietary knowledge is the differentiator. That framing keeps leaders focused on what matters—where unique data drives outcomes—and avoids over-engineering use cases that don’t need it.

When designed correctly and deployed securely, SLMs should outperform larger competitors in predictability, efficiency, and outcome success – all the areas that matter most to stakeholders. Ultimately, this benefits your business, because Willie Sutton was wrong – delivering for creating true business value is where the money really is!

Sham Arora is CTO of Tech Mahindra.