Evaluate an AI software before a team purchase
Evaluate an AI software: compare need, data, quality, operations and exit on a representative test set before buying.
By Pablo Nastar

A sales demo uses good data, a known scenario and a person who masters the product. Your team will work with incomplete requests, sensitive documents and real deadlines. To evaluate an AI software, you therefore have to test a business task, its errors, its contract and its exit before negotiating the number of licences.
The verdict is not « good » or « bad ». It specifies for which use, with which data, under which controls and until which date the tool can be chosen.
Contents
- Write the decision before the test
- Build a representative test set
- Evaluate five dimensions separately
- Calculate the full cost of use
- Decide, limit or abandon
- Frequently asked questions
Write the decision before the test
The test must answer a precise purchasing decision, not a general curiosity about AI. Describe the task, the user, the deliverable, the frequency and the effect of an error. Without that framing, each participant will judge a different promise.
Draft a usage sheet
The sheet fits on one page. It names the authorised input, the expected output, the human control, the possibly connected system and the process owner. It also specifies the current path without AI. This reference lets you compare the solution with real work, not with an ideal situation.
The CNIL offers a set of questions before using an AI system, in particular on the purpose, proportionality and affected persons. This framework does not decide for you, but it prevents reducing the evaluation to the quality of a single answer.
Define the stop conditions
Write before the test what blocks the project: forbidden data, output impossible to verify, irreversible action, lack of a suitable contract, uncontrollable cost or dependency with no export. A stop criterion formulated after failure too easily looks like a justification.
The Educasium guide on choosing a pilot use case helps select a bounded task. The AI skills audit then lets you check whether the team can really supervise the use.
Build a representative test set
A good test set contains ordinary cases, difficult cases and cases that must be refused. It is prepared before the vendor configures the demonstration.
Choose authorised examples
Use dummy or properly authorised data. Reproduce the structure of real documents without copying a client file when the framework does not allow it. Keep a reference answer or, if several answers are possible, the criteria of the business judgement.
Introduce everyday defects
Add an incomplete input, an ambiguous term, an old version, a poorly formatted document and a contradictory instruction. Also test the absence of information: should the tool ask for a clarification, flag the unknown or invent a plausible answer?
Hypothetical implementation. A company could prepare a dummy corpus of documents, one carrying an old date and another containing a contradictory instruction. The testers would compare conflict detection, citation quality and ease of correction. This scenario is not an Educasium deployment.
Evaluate five dimensions separately
A global score hides the causes of failure. Rate each dimension with evidence and a comment, then decide which dimensions are blocking for the use.
| Dimension | Question | Evidence |
|---|---|---|
| Business usefulness | does the output help the defined task? | comparison with the reference |
| Data | are the inputs and processing admissible? | contract, settings, documented flow |
| Quality | are the errors detectable and correctable? | results on ordinary and edge cases |
| Operations | does the tool fit the rights and support? | administration and export test |
| Reversibility | can you exit without losing the work? | export, deletion, return procedure |
Business quality
Ask the specialist to explain the flaw, not just to give a mark. An answer can be correct but unusable because it does not cite the source or requires too many corrections. Another can be imperfect but useful as a draft if its status stays clear.
Data and security
The France Num-CNIL sheet on choosing a generative AI solution invites you to examine the versions, the settings and data management. The ANSSI guide completes this reading on the security of the architecture. No public document exempts you from studying the vendor's precise contract.
Daily operations
Test adding and removing a user, the roles, the logs, the support, the export, version changes and deletion. A tool that is useful during the pilot can become costly if it requires invisible administration or if every incident depends on a single contact.
Calculate the full cost of use
The licence price is only one line. Add the scoping, the training, the data preparation, the verification, the administration, the connectors, the support and the possible exit.
Compare with the current path
Document the time and errors of the current process over a reasonable sample. Do not assume that every minute spent without AI is recoverable. Verification can shift the work rather than remove it.
The decision file presents the assumptions, not a guaranteed ROI. The guide on the ROI of AI training explains how to define a baseline and measure after a pilot. Before that measurement, the value remains a hypothesis.
Plan for version changes
Services evolve. Keep the model or version name, the date, the settings and the test set. A major update triggers a reassessment of the critical cases. The initial test does not hold for all future versions.
Decide, limit or abandon
The committee must be able to choose four outcomes: authorise, authorise under conditions, extend the test or stop. The refusal is not a failure of the pilot; it avoids an ill-founded purchase.
Draft a decision sheet
The sheet specifies the users, the data, the tasks, the controls, the exclusions, the owner and the review date. It also indicates the reporting procedure and the return to the previous method. Exceptions are handled separately.
Step 1: gather the raw results. Do not keep only the average.
Step 2: reread the stop criteria. A critical block does not disappear in a good overall score.
Step 3: bring together the competent functions. Business, security, data, procurement and operations take part depending on the use.
Step 4: write the conditions. Delimit the accounts, data and actions.
Step 5: prepare the exit. The team knows how to return to the previous process.
Step 6: set the review. The decision expires or is re-examined when the context changes.
Explicit absence of client experience. Educasium claims here neither a purchase made for a client nor a saving obtained. The grid is a teaching method. The verdict must come from the tests and contracts of the organisation concerned.
Limits and counter-cases
A short test does not always reveal problems of load, support or model change. A test set that is too clean favours the solution. A group made up only of enthusiastic people underestimates correction and administration.
If the use does not produce a clear decision, if the needed data cannot be used or if no one knows how to control the output, stop. Training only makes sense if the process, the tool and the responsibilities can be made consistent.
Frequently asked questions
How many tools should be tested?
Test only the solutions that pass the initial filter of need, contract and data. Multiplying products increases the work without automatically improving the decision. Two well-examined options may be enough. Also keep the current method as a reference.
Is a single score needed?
A score can help summarise, but it must not hide a blocking condition. Present the dimensions and the evidence separately. A good business result does not offset forbidden data. The committee must be able to understand the cause of the verdict.
Who takes part in the test?
Include the real users, the business owner and the functions needed for the risk: security, data, procurement or legal. Add a person unfamiliar with the tool to test learning. The vendor can explain the product, but does not validate your use. The decision remains internal.
When should you re-evaluate?
Re-evaluate after a significant change to the service, the contract, the data or the process. An incident or a new use can also trigger the review. Keep the test set to compare versions. Always date the verdict.
Buy on evidence, not on demonstration
Evaluating an AI tool means testing the task that matters, the error that worries you and the output you will need. The final file must make it possible to understand what was authorised and why.
The introduction to AI and prompting training can support a pilot by giving the testers a common method. Training eligible for OPCO/FIF PL funding subject to conditions. Qualiopi-certified organisation. To define the suitable path, contact Educasium.