How to evaluate an AI vendor: 5 questions that expose the BS

The AI services market attracts BS for a structural reason: the barrier to entry is a landing page, the buyer usually can't evaluate technical claims, and the hype gives cover to anyone confident enough. You don't need to become technical to defend yourself. You need five questions. A real vendor answers all five comfortably. A BS vendor gets vague at question one.

Question 1: "What happens when it's wrong?"

Every AI system produces wrong output sometimes. Every single one. So this question has a real answer at every legitimate shop: where humans review, what falls back to a person, what gets blocked outright.

A good answer sounds like a description of specific checkpoints: which outputs a human approves before they go out, what the system does when it isn't confident, what the customer sees when it fails.

The red flag: "it's very accurate." That's not an error-handling plan. That's a hope.

Question 2: "What exactly is custom here, and what's a wrapper?"

Most AI products are built on a handful of big models with custom work layered on top. That's a wrapper, and wrappers aren't bad. Almost everything is one, including good products. Lying about it is bad, and wrapper prices for wrapper work matter.

A good answer names the underlying model provider without flinching and tells you exactly where their own work lives: the integration, the workflow, the guardrails, the interface.

The red flag: "our proprietary AI," pressed twice, with no straight answer about what's underneath.

Question 3: "Where does my data go?"

Your customer data will flow through this system, so the vendor should be able to walk the path: which companies touch it, how long each keeps it, whether it's used for training and how that's turned off.

A good answer names the providers involved, states retention in plain terms, and points to the policy pages that back it up. You're not auditing them. You're checking whether they've ever thought about it.

The red flag: "it's all encrypted and secure." Encryption answers a question you didn't ask. Where does it go?

Question 4: "What does this cost to run, not just to build?"

AI systems have ongoing costs: model usage billed per use, maintenance, updates when providers change things. A build quote without running costs is half a quote, and the missing half arrives as a surprise.

A good answer includes a monthly estimate tied to your expected volume, and says what happens to the bill if usage doubles.

The red flag: a single one-time number and a subject change.

Question 5: "What happens if the model you built on changes or shuts off?"

The big providers retire and replace models regularly. Legitimate builders plan for it.

A good answer: how they've handled past model changes, roughly what a migration involves, and whether it's covered or billed.

The red flag: a blank look. It means they haven't been operating long enough to have lived through one, or they don't maintain what they ship.

The scorecard

Ask us the same five

That's the pitch, honestly: ask us these questions. Any vendor worth hiring will enjoy answering them, because the questions are only threatening if the answers are.