In May, Channelholic regulars may recall, I issued a plea for a vendor-neutral benchmark MSPs could use to evaluate the product claims service desk automation vendors were continually throwing at them (and me). Six months later, sadly, we still don’t have one.
We do now, however, have a small, very interesting step in that direction courtesy of Shield, the heavily funded MSP rollup I’ve written about before, including five weeks ago when the company went public with details about Forge, its custom-built “AI operating system for IT services.”
Per that post, Shield’s big-picture goal is to pair the industry’s best humans with its best computer. Logically speaking, you can’t build the industry’s best computer without help from the industry’s best AI models, and you can’t identify the industry’s best AI models without objective data to guide you.
Which brings us to HoldingsBench, an LLM leaderboard owned and operated by Shield parent Thrive Holdings that tells us both a lot and not enough about what, from a managed services standpoint, the industry’s best AI models are.
It tells us a lot in the sense that HoldingsBench offers performance, speed, and cost metrics on how well 57 closed and open source LLMs handled 52 real-world tickets that Forge’s help desk agent got wrong. It tells us not enough, though, in the sense that those metrics don’t—and can’t—pinpoint what “best” is when it comes to processing tickets.
Is best synonymous with resolution quality, for example? By that measure GPT-6 Astra with extra high reasoning, GPT-6 Astra with (merely) high reasoning, and Claude Opus 5.5 Max are the best models on the market:
They’re also merely average with respect to speed, however, and speed matters a lot in managed services, no?
“You could be the most accurate model in the world, but if you take 2x the amount of time to solve a ticket, it’s actually a much worse outcome for the end customer because they have to wait double the time,” observes Mayank Jain, Shield’s head of technology.
So clearly the fastest models, meaning Claude Sonnet 5.5, Claude Opus 5.5, and GLM 5.3 Flash, are the best according to Shield’s tests:
Except that the two Anthropic models on that list are among the highest on price, which weighs heavily on margins, and those matter a lot in managed services. So maybe lowest cost is synonymous with best for MSPs, in which case GPT-6 Luna low, medium, and high are the leaders:
Though come to think of it, how much have you really saved if you end up spending less money on a ticket that your AI fails to close, which is far more likely to happen with GPT-6 Luna than with the much more expensive GPT-6 Astra.
See how hard this is?
As Shield itself will tell you, selecting models for service desk automation involves inescapable tradeoffs among resolution quality, time, and cost with no one, easy answer because the whole question is ultimately more a matter of philosophy than science.
“We take the philosophy that quality comes first,” says Jain (pictured).
With speed a close second, he continues. Data of the kind now available on HoldingsBench helps Forge route tickets in real time to whichever model will deliver the fastest results for a given technical problem without compromising quality. Which, to further complicate matters, is constantly changing.
“We’re learning more literally every single month,” Jain says.
In part, of course, because new models appear every month. Indeed, users added some 747,000 models to Hugging Face in the first eight months of this year alone.
“With every model release, cost goes down tremendously, speed goes up, and accuracy also goes up,” Jain says. Shield adjusts the “constellation of models” it’s using at any given time every few weeks in response.
“As these new models come out, we can dynamically swap things around to be on the Pareto frontier of cost, accuracy, and time, which is where we’re all always aiming to be,” Jain says.
It’s an easier goal to realize if you’ve got the kind of money required to build a service desk automation system rather than buy one, adds Raghav Kotha, Shield’s head of strategy and growth.
“As we continue to be a research-first organization, we’re always rethinking and reevaluating our priors,” he says. “Being able to channel that almost instantaneously in the way the product shows up and the outcomes that we’re able to bring to our customers is the special sauce of the business.”
Homework for the frontier labs
Keep in mind when evaluating the pass/fail numbers on HoldingsBench that they’re not a reflection of how well AI handles service tickets writ large.
“If we’d picked a random subset of tickets, accuracy would actually have been extremely high,” Jain says, noting that Shield intentionally limited its testing sample to especially tough cases.
Partly because a saturated leaderboard full of 95% effective models wouldn’t be terribly useful and partly for a specific reason I touched on the last time I wrote about Shield: OpenAI has owned a piece of Thrive Holdings for nearly a year now, which gives Shield, in its own view, both an opportunity and responsibility to influence future models from OpenAI and others in a way that benefits everyone in managed services.
“We want these results to be a motivator to the people building frontier technology to think about IT services use cases and improve outcomes for the employees and the customers that will be served by these models,” Kotha says.
The human-computer balancing act
Performance versus speed versus cost isn’t the only help desk tradeoff Shield has to consider. There’s another as well tied to the “human-computer teaming” strategy I wrote about earlier and alluded to above. If your goal is to pair great people with a great computer, when should you use those people to handle support issues and when should you use the computer? Yet again, there are no easy answers.
“We have this opportunity to deepen the context around the optimal human-computer partnership,” Kotha (pictured) says, “and what we’re learning while we spend so much time in these real-world contexts is that it’s sort of an amorphous problem space.”
In no small part because the computer half of the equation can be unpredictable at times. “We’ve seen certain classes of models indulge in reward hacking behaviors,” Jain says by way of example, citing instances in which Shield’s help desk agent has closed a ticket even when explicitly instructed that it should always escalate issues of that kind to humans.
“Sometimes, some classes of models are actually overeager to still just go solve the ticket because that was the original task,” Jain notes.
Keeping humans in every loop isn’t the right response to such behaviors, though, given all the things those humans could be doing with the time AI saves them. Shield is experimenting with a number of them, some of which (like building AI solutions for clients) grow the top line and others of which (like doing root cause analysis of recurring technical issues) grow the bottom line. It has yet to decide which of the two options it prefers.
“We’re seeing really good early results on both,” Jain says.
BTW, shouldn’t it be ShieldBench rather than HoldingsBench?
No, actually, because Shield is owned by Thrive Holdings and Thrive Holdings owns an accounting business named Current in addition to Shield. HoldingsBench includes benchmarks for both companies, which is why I now know that the most powerful AI models can handle tough tax preparation issues at a significantly higher success rate than they can tough IT support issues.
Can’t wait to see how well AI takes on the toughest home security issues.
Don’t like your PSA? Build one instead.
As anyone at Shield will tell you, building your own tool stack is a major investment. Even in the age of Cursor and Codex, developing software takes time and money, and there are other, less tangible costs to consider as well. The benefit you get for shouldering those costs, however, is the ability to construct something perfectly aligned with your needs.
Just ask Brandon, Fla.-based MSP DataCorps Technology Solutions, which has replaced its ready-to-wear PSA solution with a home-stitched “business operating system” all its own.
“It’s very unique to DataCorps because it fits my desires, my intent as the business owner, and how I want to run the business,” says CEO Angel Rojas (pictured). It’s also made the company significantly more efficient and effective, he adds.
“Every one of our tools that we were paying money for was actually hindering our process, adding errors, adding extra time and effort, and limiting our capabilities,” Rojas says. The custom-fit tools it’s built for itself accelerate workflows, expand capabilities, and get better continuously, rather than whenever a vendor decides to add a feature you badly want to its roadmap. Rojas experiences the difference personally every time he files his company’s sales taxes, a process that used to take three hours.
“Now it takes me about five minutes a month to do it, because what we’ve built is monitoring the sales tax and the changes in the law on top of how they get applied to our invoices each time an invoice is created,” he says.
All told, DataCorps puts about four hours a day into internal software development. The skills it acquires along the way are well worth the cost though, according to Rojas, because they can be used to build custom apps for customers too.
And both rapidly and inexpensively at that, Rojas adds, thanks to AI coding tools. Indeed, DataCorps can usually have a proof-of-concept version of an end user solution up and running within a month of requirements gathering.
“We can do a trial for next to nothing and provide the client something that gets them really excited,” Rojas says.
It’s not just the demo that’s exciting either. The accounting system DataCorps built for one of its clients cut its weekly payroll process from five hours to five minutes and identified 15 people no longer with the firm but still getting checks. It also uncovered a giant pile of unpaid bills.
“They were 90 days past due on over $2 million and couldn’t figure out why they kept hitting the credit line,” Rojas says.
Projects like that yield 90 to 95 percent margins on average, according to Rojas. More importantly, he adds, they inspire the kind of loyalty only a strategic consultant can command.
“We’re not just managing technology. We’re providing technology solutions,” he says, and building a sustainable future for itself in the process.
“Here’s the reality: Everything that we’re doing today as an MSP is dying. We’re in a race to the bottom and commoditizing,” Rojas says. “Either we adapt and we grow, or we just continue to whine about margins being reduced.”
Software by an MSP for an MSP
DataCorps’s frustration with vendor-made solutions is nothing new. MSPs have been grumbling about tools for as long as there have been MSPs using tools.
What Rojas, Yaron Baitch of IgnitHQ, and Kevin Damghani have in common courtesy of AI is the ability to stop wishing for better software and start building it, faster and less expensively than their industry ancestors could dare to imagine.
“Could any dev team build a PSA a few years ago without tens of millions of dollars in upfront development costs? No,” says Damghani, CEO of ITPartners+, a mega MSP with offices in six states and the Philippines. Today, he continues, is a very different story.
“The barrier to entry is definitely a little bit lower,” Damghani (pictured) says.
Lower enough, in fact, to give him the courage to build the Swiss Army Knife managed services tool he’d long wanted but couldn’t find.
“I’ve always had this vision for a single pane of glass for an MSP, and there hasn’t been the right time to be able to do that,” he says. “I think now is the right time.”
Called Current, the system he and his team built isn’t a PSA solution so much as an overlay for PSA solutions from ConnectWise, Halo, and Kaseya, among others, that unifies CRM, project management, QBR, and dashboarding functionality.
AI-native functionality, Damghani emphasizes. The CRM module, for example, sorts through tickets and emails for signs of unmet customer needs and then converts what it finds into sales opportunities automatically. Close one of those deals and the system automatically builds a project plan based on your quote and statement of work, assigns resources, and oversees execution. Damghani’s team has some 70 projects going at any given time, and Current saves the person responsible for them an average of 90 minutes per week per project.
Based on a hunch that other MSPs will find numbers like that appealing, ITPartners+ has been selling Current commercially since September for an introductory $50 per user per month. The price will rise to $149 once Damghani signs his 100th partner, but it will stay there even as Current’s capabilities expand. ITPartners+ has service desk and procurement modules set for release this quarter, for example, and others behind them on the roadmap.
“We’re not done yet. We’ve got a lot more to go and a lot more modules to release,” Damghani says. “We’re really trying to reimagine what an MSP needs in 2026 that doesn’t look like a 20-year-old dinosaur with AI bolted on in the corner.”
If they’re successful, he adds, the product that results will be an agentic system of action that (per a prospect I discussed in this year’s very first post) turns the PSA system of record most MSPs live in today into a headless piece of backend infrastructure software.
“I wouldn’t want to be in the traditional PSA business today,” Damghani says.
A little security scariness, just in time for Halloween
I attended this year’s Black Hat conference just as OpenAI’s Hugging Face breach was making headlines. Few of the experts I interviewed at the time seemed as anxious about what that incident heralded for the future as I was.
Which I found somewhat reassuring. People who know security way, way better than I do seemed to regard LLMs like Anthropic’s Mythos as dangerous but not, per my event coverage, in some radically new way or to some radically new degree.
Maybe it’s just me, but it feels like the PHP2Shell exploit unleashed upon the world week before last has changed things. The vibe among the security experts I know before PHP2Shell was that this is bad, but we’ve been through bad before and always muscled our way through. After PHP2Shell, though, the kind of people previously worried about a mere “patch storm” are now warning of a “breachpocalypse” instead.
And PHP2Shell is only the beginning. Equally destructive exploits are certain to arrive with greater and greater frequency going forward, per Robert Johnston, chief innovation officer at N-able, during an interview on the latest episode of MSP Chat, the podcast I co-host.
“These technologies are really good at finding vulnerabilities. They can step through billions, trillions, of lines of code in a very short amount of time, and they don’t miss anything,” he says. As a result, he continues:
You are going to get a parabolic increase in the finding of vulnerabilities, which will lead to a parabolic increase in the exploitation of vulnerabilities. And we’re going to have to somehow eke out a parabolic increase in the way we patch and secure vulnerabilities. And that’s like the open question that everyone is talking about now.
The whole informed, insightful, and deeply sobering interview is located here. Other enlightening but less frightening thought leader conversations are available here.
Over on Business of Tech
Most of you have read my recent thoughts about Pax8’s new persistent agent, Rewst’s new RPA platform, and Neo Agent’s credit-based pricing scheme. Tune in here if you’d like to hear me trade thoughts about those and other topics with Business of Tech host Dave Sobel, as I did on the latest live episode of his show.
Let’s talk
When not writing this newsletter, I’m chief analyst at Channel Mastered, a consultancy that helps vendors build, grow, and optimize thriving MSP channels. Wrestling with a channel challenge? Hit me up here: rich@channelmastered.com.
Also worth noting
Expect lots more on this at IT Nation Connect next week: ConnectWise has introduced agentic RMM functionality designed to autonomously investigate and close tickets.
Move faster, Microsoft partners. Starting November 2, you have five calendar days to accept or reject SMB customer referrals versus the previous 14.
The latest updates to Cisco’s partner program include new customer experience and AI-ready technologies designed to add revenue opportunities across the opportunity lifecycle.
Neo Agent’s AI-native service desk automation solution, covered here last week, is now available through the Pax8 Marketplace.
1Stream’s AI-powered call handling functionality now lets NinjaOne PSA users turn inbound support calls into tickets automatically.
N-able’s AI-accelerated vulnerability and patch management functionality is designed to identify security exposures, prioritize remediation, and automate patching.
GravityZone Exposure Management, from Bitdefender, is designed to identify and prioritize vulnerabilities, misconfigurations, and excessive privileges.
Fortinet’s new FortiGuard Inline AI Security Service aims to help users block malicious prompts, sensitive-data exposure, and other AI-specific threats.
Cortex Cloud SaaS Security, from Palo Alto Networks, offers unified visibility, threat detection, and protection across SaaS applications and cloud environments.
ESET’s new NeoCortex AI model features a “never-go-rogue” architecture, the company says, and costs 200 times less than comparable systems to train.
Cato AI Endpoint, from Cato Networks, is designed to extend visibility and security controls to AI applications and agents operating outside web browsers.
Flex, from Proofpoint, is a new purchasing program designed to help customers adapt security investments as their requirements evolve.
LUMINATE, from DNSFilter, is a unified security platform combining DNS protection, user behavior analytics, AI governance, and privacy controls.
The latest enhancements to JumpCloud’s continuous compliance capabilities are designed to help organizations automate evidence collection and monitor security controls in the agentic era.
Newly added preemptive security functionality in SimSpace’s software helps users guard against active nation-state threats before attacks occur.
Action1’s new endpoint takeover prevention architecture is designed to stop attackers from weaponizing the company’s endpoint management functionality.
Pulse AI Resolve, from ControlUp, is an AI-powered IT investigation and remediation tool.
The new SMB-focused additions to TD SYNNEX’s Destination AI program are designed to help partners develop, package, and scale repeatable AI services.
Also new from TD SYNNEX: partners can now offer self-serve Cisco purchasing through white label storefronts thanks to new real-time integration with Cisco Commerce Workspace.
Legal AI vendor Clio has acquired Page to help legal pros better anticipate and respond to regulatory and legislative changes.
Dell has expanded its cybersecurity and resilience portfolio across widely used and frequently targeted AI environments, identity systems, and infrastructure.
Dakota Ferguson is the new CEO of Lyra Technology Group, Evergreen’s mega mega MSP platform.
Dan Fletcher is the new CFO at Barracuda.