The default of routing everything to the biggest model is ending
Speaking at the Evident AI Symposium in New York on October 22, Hari Gopalkrishnan, Bank of America's chief technology and information officer, described a routing layer called Orchestra that sends each incoming query to the most cost-effective model available rather than defaulting every request to the most capable, and most expensive, frontier model. His reasoning was direct: "there are simpler classifier models that do very easy decision things that you don't need a large language model for." That is a correction to an assumption a lot of enterprise AI deployments made by default over the past two years, that bigger and more general is automatically better, when in production at scale the cost difference between routing every query to a frontier model versus a lightweight classifier compounds fast.
Gopalkrishnan also framed the vendor-independence angle plainly: "We didn't want to be beholden to any given model." A routing architecture does two things at once. It controls cost by matching task complexity to model size, and it protects the bank from being locked into a single model provider's pricing or roadmap, which is a real strategic risk for any enterprise that has not built an abstraction layer between its applications and the specific model underneath them.
Capital One's numbers make the case harder to argue with
Prem Natarajan, Capital One's chief enterprise AI officer and chief scientist, described a similar move with harder numbers attached. Capital One distilled its open-weight models, fine-tuned on proprietary data, into smaller models that now power Chat Concierge, a multiagent application that helps car buyers through the purchasing process. The result: latency cut fivefold and dealership engagement up 50%. Those are the kind of production metrics that settle an internal debate about whether a smaller model is a compromise, in this case it clearly was not.
Natarajan also connected this back to Capital One's longer AI history: the company's early move to a cloud tech stack and years of machine learning experience had already taught it the value of data discipline, and he said generative AI and agents raise the bar for data readiness further still. When architecture needs to change, he said, Capital One rebuilds foundations from the ground up rather than patching around the gap, which is a more disciplined posture than most enterprises can claim when a new model generation arrives.
Spending more does not mean spending less carefully
Bank of America plans to double its AI budget next year, which on its own might sound like evidence that cost discipline is losing out to growth ambition inside the organization. Gopalkrishnan tied the increase specifically to handling more advanced conversations with Erica, the bank's AI-enabled financial assistant, rather than to a general expansion of frontier model usage across every function the bank runs. The budget is growing because the bank wants to do measurably more with AI across a widening set of use cases, and the routing strategy is precisely what makes that growth affordable in the first place rather than something working against it.
That distinction, between spending growth tied to expanded scope versus spending growth caused by inefficient architecture, is exactly the kind of nuance that gets lost in a budget line item. A CIO reporting a doubled AI budget to the board without being able to show the routing discipline underneath it risks the number being read as waste rather than ambition, even when the underlying math is sound.
The vendor market is catching up to the pattern
Snowflake has already added dynamic model routing to its Cortex AI Gateway, automatically selecting an appropriate model for each query, which means this pattern is becoming available as an off-the-shelf capability rather than something only banks with large internal AI teams can build. That matters for the large majority of enterprises that do not have the engineering capacity Bank of America and Capital One have invested in their own routing and distillation infrastructure.
If model routing becomes a standard feature across major AI platforms over the next year, the practical effect is that the cost-discipline advantage these banks built by hand becomes table stakes rather than a differentiator, which should push CIOs who have not yet asked this question to ask it now: is your organization currently sending every AI query, regardless of complexity, to the most expensive model available by default, and if so, what is that costing against the alternative.
Guardrails are showing up alongside the cost controls
Gopalkrishnan also described Bank of America's 2027 strategy as including tightly controlled guardrails: agents must stay within their allotted runtime and cannot autonomously complete multiple tasks in a chain without a human checkpoint somewhere along the way. He emphasized telemetry and feedback loops as areas the bank is still actively developing, a candid admission that the observability layer is not fully finished even as the routing and budget decisions keep moving forward on their own separate timeline.
That sequencing, cost architecture and governance guardrails developing in parallel rather than governance arriving as an afterthought once something has already gone wrong, is the healthier pattern compared with deployments that scale first and bolt on oversight later. It is also a direct, practical answer to the regulatory and incident-driven pressure building elsewhere in the industry around exactly what autonomous agents should be allowed to do without a human somewhere in the loop watching for trouble.
What this means for the roadmap
The actionable question for any enterprise technology leader is whether your AI architecture currently has a routing layer at all, or whether every request, regardless of complexity, defaults to whichever model your team integrated first. If the answer is the latter, treat that as a major cost lever sitting unused, the same gap Bank of America and Capital One identified and fixed before the cost curve got away from them.
The second question, equally important, is whether your governance guardrails are being built alongside your cost architecture right now or are still sitting on a roadmap as a planned future phase. Bank of America's answer, developing both at once even while openly admitting the observability piece is unfinished, is a more honest and more defensible position to take with a board than claiming either one is complete when it plainly is not yet.
