I sat through a lot of vendor demos back when I was a CIO. Lately I've been on the other side of the table, pitching instead of buying, and something has changed that every healthcare operator should understand before they sign anything.

The demos are all good now.

The voice sounds natural. The AI handles the interruption, catches the insurance question, books the appointment. Eighteen months ago you could separate the serious products from the weak ones in about ninety seconds, because the weak ones sounded like a phone tree from 2009. That tell is gone. The language models improved for everyone at once, and voice quality has converged toward something close to parity.

This is good for patients. It is not good for the operator trying to make a decision, because the thing that used to do most of the sorting no longer sorts anything.

There is also a second problem, and it is bigger. A demo is a controlled environment. Clean inputs, a single happy-path scenario, nothing else competing for the system's attention. Real life is that same product colliding with your EHR, your scheduling rules across dozens of locations, your insurance panels, and thousands of live calls hitting it at the same time. Two platforms can look identical on a screen and behave nothing alike once integration and scale enter the picture. Latency creeps in. Edge cases pile up. A rule that worked for one clinic quietly breaks for another. You cannot see any of that in a demo, because a demo is designed to keep it out.

What converged, and what didn't

Conversation quality converged. The workflow underneath it did not.

A natural-sounding call is roughly the first twenty percent of the job. The other eighty percent is everything the demo never shows you: whether the appointment survives the handoff into the EHR, whether the scheduling rules for a specific provider at a specific location with a specific insurance panel are actually encoded, whether the system knows what to do when a referral arrives by fax half-legible, whether anyone is accountable when the numbers come in soft three months later.

None of that fits in a twenty-minute demo. All of it determines whether you get a return.

So the evaluation has to move past the part that impresses and toward the part that pays. Here are the three questions I would ask, and the reasons most vendors struggle with them.

1. Does the appointment actually land in the EHR?

There is a meaningful difference between "we contacted the patient" and "the patient is on the schedule, confirmed, written back to your system of record." Plenty of platforms stop at the conversation and report activity. The value leaks out in the gap.

Consider referrals. In the practices we've worked with, an average of 38% of inbound referrals never result in a booked appointment. These are patients a physician already sent you. The demand is not the problem. The follow-through is. A platform that reads the fax, creates the record, reaches the patient, and books them, all the way through to a confirmed slot in the EHR, is solving a different problem than one that simply places an outbound call and logs the attempt.

Ask to see the last mile, not the first minute. Ask what percentage of interactions end in a confirmed, EHR-written appointment, not a "successful contact."

2. How does the vendor get paid?

Pricing is the most honest signal a vendor sends, because it reveals what they are actually confident in.

Most healthcare technology is priced on activity: seats, calls placed, messages sent, reports generated. That model pays the vendor whether or not anything happened for the practice. If a vendor will only price on activity and won't stand behind booked appointments or resolved interactions, that reluctance is information. It usually means they aren't sure the outcomes will show up.

The alternative is to tie the vendor's revenue to yours. When that alignment exists, everything downstream changes: how the product gets built, how implementation is handled, how the team responds when something breaks. You want a partner whose incentive is identical to yours, not one who gets paid for effort.

3. Has anyone on their team actually run a schedule?

This is the one most operators underweight, and it is the one I trust most.

Building software for healthcare operations is not the same as having lived them. The difference shows up in the product. Someone who has sat in the operations chair knows that "just book the appointment" is never just booking the appointment. They know a Mohs follow-up is not a cosmetic consult, that a template can break silently and cost you a week of revenue before anyone notices, that the person who really understands your scheduling logic is the front desk lead who has been there fifteen years and never wrote any of it down.

Operator DNA is hard to fake and impossible to demo. Ask who on the team has run healthcare operations, not just built software for them. The answer tells you whether the product was designed around your reality or around an engineer's assumption of it.

What the outcomes look like when the eighty percent is handled

When the workflow underneath is built well, the results are not subtle. One enterprise practice we worked with consolidated two separate outreach vendors onto a single platform and, within the first weeks, rebooked 73% more cancelled appointments, re-engaged 76% more no-show patients, and booked 13% more total appointments than the prior intake system was generating. Across a national specialty deployment, targeted outbound outreach produced more than 200,000 booked appointments and roughly $40M in revenue impact over the first year, most of it revenue that would otherwise have leaked out through cancellations, no-shows, and unfilled slots.

Those numbers did not come from a better-sounding voice. They came from the unglamorous parts: the EHR integration, the scheduling logic, the accountability for outcomes. The parts the demo skips.

The takeaway

Voice quality is table stakes now, and it will only get more so. If two vendors sound equally good, you have learned almost nothing, because you tested the part that has already converged.

Move the evaluation to the part that hasn't. Does the appointment land in the EHR? Will the vendor get paid on outcomes? Has anyone on the team actually run the operation you're asking them to automate?

The demo shows you the twenty percent that's easy. The eighty percent that's hard is where your return lives. Ask about that.