How to Evaluate an AI Vendor (Without Getting Burned)

The AI vendor market right now is a mess. Every IT consultancy, digital agency, and one-person shop has added "AI" to their website. Some of them are brilliant. Some of them are repackaging ChatGPT with a logo on it and calling it a platform. The problem is, from the outside, they all sound the same.

We're going to give you a practical framework for telling the difference — before you sign anything, before you hand over a deposit, and before you waste three months on a project that was never going to work. Yes, we're an AI vendor ourselves, so take this with a grain of salt. But the advice is genuine. A good vendor won't be threatened by informed customers. We'd rather compete on merit than hope you don't ask the right questions.

Red flags: walk away

These aren't nitpicks. If you spot any of these, it's a serious warning sign. One on its own might be forgivable. Two or more? Find someone else.

1. They can't explain what the AI actually does in plain language

If a vendor can't describe how their system works without resorting to jargon — "neural networks," "transformer architecture," "proprietary algorithms" — they either don't understand it themselves or they're hiding something behind complexity. A competent developer can explain any AI system in terms a business owner would understand. If they can't, or won't, that's a problem.

2. They want a 6-month "discovery phase" before building anything

Discovery is real. Understanding your business before building is important. But six months of it? That's a consulting engagement disguised as a software project. Most small business AI tools can be scoped in a few conversations and built in weeks. If someone needs half a year just to figure out what to build, they're either padding the invoice or they don't know how to build it. A clear brief should get you to a proposal within days, not quarters. We've written a guide to writing a good brief that covers exactly how to shortcut this process.

3. They can't show a working demo within two weeks

This one sorts the builders from the talkers faster than anything else. If a vendor has genuine capability, they should be able to produce a rough working demo — not polished, not production-ready, but functional — within two weeks of understanding your problem. If they need months before you see anything working, ask yourself why.

4. Per-seat pricing for internal tools

This is a SaaS pricing model bolted onto custom work, and it makes no sense for internal tools. If a vendor builds you an AI system that your own staff use, why should adding a fourth or fifth user cost extra? The marginal cost of another user on an internal tool is effectively zero. Per-seat pricing is designed to scale revenue with your headcount, not to reflect actual costs. For internal tools, insist on flat pricing or usage-based billing.

5. They won't let you own the code or data

This is a dealbreaker. If you're paying for custom development, you should own what gets built. That includes the source code, any models fine-tuned on your data, and — obviously — your data itself. Some vendors will build on proprietary platforms specifically so you can't leave. If they won't put code and data ownership in the contract, you're not buying a tool. You're renting a dependency.

6. They promise "90% accuracy" without defining what that means

Accuracy figures without context are meaningless. Ninety percent accuracy at what? On what data? Measured how? A spam filter that's 90% accurate still lets one in ten junk emails through. An AI receptionist that's 90% accurate mishandles one in ten calls. The number means nothing without a clear definition of the task, the test data, and the failure modes. If a vendor throws accuracy numbers at you in a sales pitch without this context, they're selling confidence, not competence.

7. The sales pitch mentions "blockchain" or "metaverse" alongside AI

This one sounds flippant, but it's a genuine signal. Companies that chase every technology buzzword tend to be shallow in all of them. You want a vendor who's deeply focused on solving real problems with AI — not one who pivoted from crypto last year and will pivot to quantum computing next year.

Green flags: good signs

These are the things we'd want to see if we were on the buying side of the table. None of them guarantee a great outcome, but they dramatically improve your odds.

1. They ask about your problem before pitching a solution

A good vendor's first question should be "what's not working?" — not "here's what we can build you." If someone leads with their product instead of your problem, they're looking for a place to install their hammer, not trying to figure out whether you need a hammer at all.

2. They start small — one tool, one workflow

The best AI projects start with a single, well-defined problem. Automate one workflow. Prove it works. Then expand. A vendor who suggests building five things at once is either overconfident or padding the quote. Look for someone who says "let's nail this one thing first" — that restraint is a sign of experience.

3. They use their own tools in their own business

This is the ultimate credibility test. If a vendor builds AI tools for others but doesn't use AI in their own operations, ask why. We run our own AI systems in our own practice — email automation, voice tools, document processing. We hit the bugs before our clients do. We know what works in a real office with real staff, not just in a demo environment.

4. Transparent pricing with no hidden fees

You should know exactly what you're paying for before work begins. Fixed project pricing or clear hourly rates, documented in writing. No surprise "integration fees," no "platform licensing" that appears on the second invoice. If you want to understand what custom AI actually costs for a small business, we've published a detailed cost breakdown with real numbers from real projects.

5. They let you walk away with everything they built

Code, data, documentation, deployment instructions. If the relationship ends, you should be able to hand everything to another developer and keep running. A vendor who builds with this in mind is confident in their work. A vendor who locks you in is betting that switching costs will keep you around longer than quality would.

6. They can show you similar projects they've completed

Not a generic portfolio of logos — actual examples of problems they've solved that are similar to yours. What was the brief? What did they build? What were the results? If they can walk you through a past project with specifics, they're probably telling the truth. If everything is "confidential," ask for anonymised case studies at minimum.

7. They talk about limitations, not just capabilities

Every AI system has limitations. Speech-to-text struggles with heavy accents. Language models hallucinate. Classification systems need clean training data. A vendor who tells you what their tools can't do is being honest with you. A vendor who only talks about what's possible is selling you a fantasy.

Five questions to ask before you sign

These questions cut through marketing and force specific answers. Ask all five. Pay attention to how they respond as much as what they say.

"Can I see a demo relevant to my industry?"

A generic chatbot demo proves nothing. You want to see something that relates to your actual use case — even if it's rough around the edges. The willingness to build a quick, tailored demo tells you more about a vendor's capability than any slide deck.

"What happens to my data if we stop working together?"

The answer should be simple: you get it all back, in a usable format, within a reasonable timeframe. If the answer involves caveats, fees, or vague timelines, that's a red flag. Your data is your data. Full stop.

"What does ongoing maintenance cost?"

Every AI system needs some maintenance. Models drift. APIs change. New edge cases appear. A good vendor will be upfront about ongoing costs — whether that's a monthly retainer, a support package, or hourly rates for ad-hoc fixes. If they tell you the system will "just run" with no maintenance, they're either naive or dishonest.

"Who owns the code?"

You want this in writing, in the contract, before work begins. The answer should be unambiguous: you own it. Some vendors use open-source frameworks underneath, which is fine — those components have their own licences. But the custom code, the integrations, the configuration specific to your business? That's yours.

"What's your smallest project to date?"

This question reveals a lot. A vendor whose smallest project was $100,000 probably isn't the right fit for a $5,000 job — their processes, overhead, and expectations are calibrated for enterprise work. You want someone who's comfortable building small, delivering fast, and scaling up only when it makes sense. The smallest project number tells you whether they'll treat your budget as a real project or an afterthought.

A note on honesty

We're not going to pretend this post is entirely selfless. We tick every green flag on the list. We ask about problems first. We start small. We use our own tools every day. We hand over the code. We talk about what doesn't work as readily as what does. Writing a guide that describes our own approach and calling it "how to evaluate a vendor" is about as self-serving as it gets.

But that's exactly why we're comfortable publishing it. If you use this framework and end up choosing someone else — someone who also ticks every green flag and happens to be a better fit for your project — that's a good outcome. You'll still end up with a better vendor than if you'd gone in blind. And if this guide helps you dodge a bad engagement that would have cost you $20,000 and six months of frustration, we'd consider that a win even if we never hear from you.

The AI market needs more informed buyers. Informed buyers raise the bar for everyone.

Ready to put this to the test?

Take this list to your next vendor conversation. Ask the five questions. Watch for the red flags. Look for the green ones. And if you'd like to see how we measure up, get in touch. We'll answer every question on this page — and we'll be honest about the ones where the answer isn't what you want to hear.