Vendor evaluation usually fails in the same way: the buyer asks about capability and the vendor answers about capability, and both leave the call knowing nothing they did not know before. Capability questions are answerable by anyone with a marketing team. Evidence questions are not.
Here is what to ask instead, and what the answers tell you.
Five questions about the work
“Walk me through a system you put into production, including what broke.”
Anyone can describe a build. Only teams who operated a system can describe its failure modes. Listen for specifics: retrieval quality degrading as the corpus grew, inference costs scaling non-linearly with traffic, an evaluation set that stopped correlating with user outcomes. A vendor who has never seen a system fail has never run one.
“What did you tell a client not to build?”
Vendors who only agree are selling hours. The answer should be concrete and recent.
“Who exactly will work on this, and can I interview them?”
Named engineers, this week. A refusal here usually means the team will be assembled after signature, and assembled teams are junior-weighted more often than not.
“What is your evaluation approach before we go live?”
For anything involving language models, the answer should include a held-out set, a defined quality bar, and a mechanism for catching regression after deployment. “We test it thoroughly” is not an answer.
“What happens to the code and the knowledge when we stop?”
Repository ownership, documentation standard, handover process. Ask it early, because the answer changes the contract.
Three questions about the arrangement
- How do you price a change in scope? Fixed-scope vendors make their margin here. Get the mechanism in writing before you need it.
- What is the notice period, both directions? Asymmetric notice tells you how the vendor thinks about the relationship.
- Where does our data sit, and under whose jurisdiction? If you process personal data under GDPR, an engineer inside the EU needs no transfer mechanism. Outside it, you need Standard Contractual Clauses, a transfer impact assessment, and somebody in your organisation who owns both. Vendors who wave this away have not read it.
Answers that should end the call
- “We can start Monday with a full team.” Nobody has five idle senior AI engineers unless something went wrong elsewhere.
- A rate that undercuts the market by more than 40%. Check the rate benchmark. A rate far below the band means either a junior wearing a senior title, or a subcontracting chain you have not been told about.
- No named engineers at proposal stage. You are buying a promise to recruit.
- A demo instead of a repository. Demos are built to survive a demo.
- Certifications used as the primary evidence. Partnership badges describe a commercial relationship with a cloud vendor, not delivery quality.
The test that beats all the questions
Give the shortlist a small piece of real work. Two to five days, paid, with your actual data and your actual constraints. Ask for a repository at the end, not a deck.
This costs a few thousand euro and resolves more than three months of proposal review. What you learn:
- Whether they ask clarifying questions before writing code, which is the single strongest predictor of seniority
- Whether their code reads like the surrounding code or like a greenfield project
- Whether they tell you the requirement was wrong when it was wrong
- How they behave when something in your environment does not work
Most vendors will decline. That is itself the result.
What we get wrong
For balance, the failure modes on our side of the table. Staff augmentation vendors, including us, are weakest when:
- You have no technical leadership to direct the work. Augmented engineers execute well against direction and drift without it.
- You need someone to sign for the outcome rather than supply the capacity.
- The engagement is under a month, where the ramp cost dominates.
The sourcing model comparison covers when a different model fits better.
The short version
Ask for evidence, not capability. Ask for names, not roles. Ask what broke. Then pay for a trial rather than reading another proposal.
If you want to run that call with us, book 30 minutes and bring the hardest question on this page.