Questions to ask a Kansas City AI vendor
Why the demo tells you so little
Every AI automation vendor has a demo, and the demo usually works. It runs on clean sample data, with no exceptions, no old systems and nobody on your team asking what happens when the answer is wrong. That is fine for a first look. It tells you very little about how the vendor behaves once the work meets your real data.
The questions below are meant to show that difference. They are not trick questions. A vendor who has put automation into a small business before will answer most of them quickly, often with a story about something that went wrong. A vendor who mainly sells will steer you back to the demo or the roadmap.
If you are still deciding which workflow to automate, start with how to choose the first workflow. This guide assumes you already have a workflow in mind and two or three vendors on a shortlist.
1. Can you run it on one of our workflows?
Ask for a working run on a real task from your business, using a sample of your own data, before any contract. Pick something ordinary: a batch of intake emails, last month's invoices, a week of help desk tickets.
Good signs:
- They ask how the workflow runs today, who owns it and where the exceptions go.
- They tell you plainly what the automation will not handle.
- They can describe a past project of a similar size, including what changed from the original scope.
Be careful with references from much larger companies. A rollout that needed a full time internal IT team tells you little if you do not have one.
2. What happens when it gets something wrong?
Every automated workflow meets cases it was not built for. The worst failures are quiet ones: a plausible but wrong answer that nobody catches until it has moved into an invoice, a customer email or a schedule.
The NIST AI Risk Management Framework is a voluntary framework from the U.S. National Institute of Standards and Technology [S2]. It treats this as something to plan for, not something to discover later. It calls for contingency processes for failures or incidents in third party AI systems deemed high risk, and for post-deployment monitoring that covers appeal and override, incident response and recovery [S1].
Ask the vendor to walk you through the path from "the automation got it wrong" to "it is fixed". Who notices? Who on your team gets the item? Can a person stop or override the automation, and how fast? What happens to the record while it waits for review?
Good answers name a mechanism and a person. "We flag it for review" is not an answer until you know who reviews it and when. Ask how many hours a week your staff should expect to spend on flagged items in the first few months. If the answer is "almost none", ask how they measured that.
3. Where does our data go?
Get clear written answers to four questions before you sign:
- What data does the system need, and what can it work without?
- Where is the data stored and processed, and who at the vendor can see it?
- Is any of our data used to train or improve models, theirs or anyone else's?
- What happens to our data when the contract ends, and how do we get it back?
Many AI products are built on models and services from other companies. NIST notes that the organization that developed an AI system may not be transparent about the risk metrics or methods it used [S1]. So ask the vendor to name the underlying providers and explain what each one receives.
If you work under HIPAA, a client data agreement or cyber insurance requirements, ask for the relevant documents now, not after signing. A vendor who cannot answer the data location question in the first meeting is not automatically a bad choice. It does mean you will probably be the first customer to make them think it through, and that takes time.
4. Who supports us locally, and who comes on site?
For a small business in the Kansas City area, this question often decides the shortlist. Remote setup works for plenty of software. AI automation that touches your phones, your network, your line of business software or your staff's daily routine often needs someone who can sit with the team, watch how the work is really done and fix access problems in person.
Ask:
- Who does the setup, and are they employees or subcontractors?
- Where are the people who will support us after launch, and can they come on site when needed?
- What are the support hours and response commitments, in writing?
- Who holds the admin accounts and API keys for the automation, and do we get copies?
- Who do we call when the automation breaks because something else changed, such as a password policy, a firewall rule or a software update?
That last question matters more than it sounds, because a lot of automation problems turn out to be IT problems. If your AI vendor and your IT provider are different companies, ask how they will work together and who owns the problem when each one says it belongs to the other. If you have no IT provider at all, that gap is worth closing first. Our managed IT services page explains how we run IT and automation under one operating model.
5. What does done mean, in writing?
The most useful document in an AI automation project is a short written definition of what working means. Without one, the vendor can call the project a success while your staff still does the work by hand around the gaps.
Ask the vendor to propose measurable acceptance criteria for your workflow before the contract is signed. For example: the share of items handled without a person, the error rate on a sample your team checks, how long a flagged item takes to resolve, and how long those numbers must hold after launch. Use numbers you can check yourself, against a baseline you measured before the project started.
Ask for the timeline as a range, and ask what made their past projects run long. Common causes are integrations that turn out harder than expected, messy source data, a security review that takes longer than planned and a project owner who changes jobs. A vendor who has done this before will tell you which of those risks is most likely for you and what you can do now to reduce it.
NIST's Cybersecurity Framework makes a related point about any supplier. Requirements belong in contracts and other agreements, due diligence comes before the relationship starts, and plans should cover what happens after the service agreement ends [S3]. All of that applies to an AI vendor too.
What to save for the second meeting
You do not need every question in the first call. Use the first conversation for questions 1 and 2. If the answers are specific, bring your operations lead and whoever handles IT or compliance to a second meeting about data, support and acceptance criteria.
The pattern to look for is detail. Vendors who have deployed before talk about problems: what broke, what they changed and what they would do differently. A polished demo cannot give you that.
Where Tensor Garden fits
These are fair questions to ask us too. Our AI automation work handles routine tasks inside a written boundary, with a person on anything outside it. For a one page checklist version of this guide, see how to choose an AI automation partner in Kansas City. If you have not picked the workflow yet, compare two workflows in about five minutes and leave with one first move.
Frequently asked questions
What questions should a Kansas City small business ask an AI automation vendor?
Ask for a run on one of your own workflows before any contract, a step by step account of what happens when the automation gets something wrong, written answers on where your data is stored and whether it trains any model, who supports you after launch and whether they can come on site, and measurable acceptance criteria agreed before signing.
Does an AI automation vendor need to offer local or on-site support?
Not always, but it helps when the automation touches phones, networks, line of business software or daily staff routines. Ask who does the setup, whether they are employees or subcontractors, what the support hours and response commitments are in writing, and who you call when an IT change breaks the automation.
How can I tell if an AI vendor has real deployment experience?
Ask for a completed project of similar size to yours, past go-live. A vendor with real experience names the process, explains what changed from the original scope and tells you what the client had to do on its side. A vendor without it describes the demo or offers a reference that has not gone live yet.
What should an AI automation contract include?
Written acceptance criteria tied to your workflow, a timeline stated as a range, data handling and retention terms, who holds admin accounts and keys, support hours and response commitments, and what happens to your data and access when the agreement ends.