Two employees upload the same customer feedback workbook, ask an AI assistant the same question, and get back different scores. A complaint about response delay comes back "high priority" in one run and "medium" in another. A product issue tops the list in one analysis and sits second in the next. Nothing in the spreadsheet changed. Only the person running the analysis changed.
This does not automatically mean the tool is broken. Large language models are probabilistic systems, not rule engines — when instructions are broad or subjective, the model has to interpret what "good" looks like, and different users, contexts, and phrasing can legitimately produce different outputs. For brainstorming or creative work, that variation is a feature. For a score that drives customer escalation, compliance triage, or resource allocation, it's a governance problem, and it needs a governance answer.
The answer is not to stop using AI. It's to stop asking AI to do a job that requires deterministic rules.
Why the Same Data Produces Different Outputs
LLMs generate responses by predicting likely continuations from a prompt and its surrounding context. Even with generation settings tuned to reduce randomness, reproducibility isn't guaranteed across every commercial or enterprise environment. Research on LLM reproducibility backs this up: studies on deterministic settings have found that outputs and accuracy can still vary across repeated runs, and separate work on numerical sources of nondeterminism has shown that infrastructure details like batch size and GPU configuration can shift generated responses even under otherwise controlled conditions.
In everyday business use, the cause is usually simpler than infrastructure. The prompt is ambiguous. The score definitions were never written down. The user has different context or custom instructions loaded. The conversation history is nudging interpretation one way or another. Or the task itself is inherently subjective.
Take an instruction as ordinary as "score customer sentiment from 1 to 10." It sounds clear and is actually full of gaps. Does 10 mean highly positive sentiment, or highest severity? Does sarcasm count as negative? Do repeated complaints push the score up? Is sentiment separate from business impact? Whatever the model doesn't get told, it fills in on its own — and it won't fill the gap the same way twice.
Where Variability Is Acceptable — and Where It Isn't
Not every inconsistency is a problem worth solving. In brainstorming, research synthesis, and marketing ideation, different outputs on different runs are useful — a model that says the same thing every time isn't exploring the space, it's rehearsing.
Variability becomes a business risk the moment the output feeds an operational decision: risk scoring, customer escalation, compliance triage, complaint prioritisation, audit review, incident severity, anything employee-facing. The test is simple — does this output need to be defensible? If the answer is yes, unconstrained AI judgement isn't the right tool. The organisation needs an approved decision framework, and the AI needs to operate inside it rather than invent it at runtime.
The Operating Model: Three Layers, Not One
A robust workflow separates the work instead of asking one model to do all of it:
- The AI layer reads ambiguous text and extracts structure — themes, categories, evidence.
- The rules layer — Excel formulas, Power Query, Power BI, SQL, workflow rules — calculates the actual score from that structure.
- The human layer reviews exceptions and signs off on decisions the rules flag as uncertain or high-impact.
This doesn't remove AI from the process. It removes AI from the one part of the process — final scoring — where its probabilistic nature stops being an asset and starts being a liability. The AI still reads, summarises, classifies, and clusters faster than any manual review could. It just no longer gets to decide, silently and differently each time, what the numbers mean. The result is a process where similar inputs get scored in similar ways, the assumptions behind every score are visible, and a business owner can actually explain how a decision was reached — to a customer, an auditor, or a board.
A Practical Framework for Customer Feedback Analysis
The same structure adapts to customer service, contact centre, complaints, training feedback, employee pulse surveys, or operational incident review:
- Define categories — response time, product quality, billing, staff behaviour, digital experience, policy/process issue.
- Separate sentiment from severity. Sentiment is how the customer feels; severity is the business or operational impact. A customer can sound calm while reporting a serious defect, or sound furious about something low-impact — conflating the two loses information either way.
- Define explicit score anchors. A five-point scale is usually easier to govern than a ten-point one, because every level can actually be described in words a reviewer would agree on.
- Calculate priority with a transparent formula — for example, 40% severity, 25% sentiment impact, 20% recurrence, 15% strategic value. The weights are a business decision made in advance, not something the model works out on the fly.
- Require evidence. Every classification carries a short quote or row reference back to the source data, so a reviewer can validate the result in seconds rather than take it on faith.
Design principle: don't ask the AI to invent the final score. Ask it to classify each dimension against the approved rubric, then let a formula calculate the result.
Illustrative Scoring Rubric
| Dimension | 1 = Low | 3 = Moderate | 5 = High |
|---|---|---|---|
| Sentiment impact | Positive or neutral tone | Dissatisfaction but not severe | Strong negative emotion, frustration, loss of trust |
| Business severity | Minor inconvenience or suggestion | Service failure affecting experience | Revenue, churn, compliance, or safety implication |
| Frequency | Isolated comment | Recurring theme across several comments | Repeated pattern affecting many customers |
| Urgency | Can be handled in normal cycle | Needs planned follow-up | Requires immediate escalation |
| Evidence quality | Weak or unclear wording | Some supporting detail | Specific, actionable evidence in the comment |
Prompt Design: From Broad Instruction to Governed Instruction
Weak prompt: "Analyse this feedback and tell me the priority." Convenient to write, and it hands the model far too much discretion — every ambiguity above gets resolved silently, differently, per run.
Better prompt: "Classify each feedback row using the approved rubric below. Do not create new scoring criteria. For each row, return category, sentiment score, severity score, urgency score, evidence quote, and confidence level. Leave the final priority calculation to the formula in the workbook."
The improved version doesn't ask the model to do less work — it asks it to do a narrower, better-specified job: classify against a fixed standard, cite evidence, flag its own confidence, and stop there. Scoring stays outside the model, where it's auditable.
Recommended Enterprise Controls
- Approved rubric — one organisation-approved scoring framework per recurring use case.
- Prompt template — a standard, published prompt so users aren't each inventing their own criteria.
- Output schema — fixed columns: category, sentiment, severity, urgency, evidence, confidence.
- Deterministic calculation — formulas, scripts, or BI rules compute the final score, not the model.
- Exception review — low-confidence or high-impact cases route to a human.
- Version control — record the prompt version, rubric version, and workbook version behind every analysis.
- Sampling audit — periodically check AI classifications against human-reviewed samples.
- User training — AI is the analytical assistant; it is not the accountable decision-maker.
Getting There: A Five-Phase Roadmap
- Identify recurring analysis workflows where inconsistent scoring could touch customers, compliance, reporting, or operational decisions.
- Convert tacit judgement into explicit rubrics — interview business owners and your strongest reviewers to document how they actually judge severity, urgency, and priority today.
- Build a standard workbook or workflow — structured input columns, protected formulas, a fixed output schema.
- Run parallel testing — compare AI-assisted classification against historical human decisions and refine the rubric before rollout.
- Operationalise with governance — named ownership, version control, audit sampling, and a defined exception path.
Four Misconceptions Worth Retiring
"If the AI gives different results, it's unreliable." More precisely: unconstrained AI is unsuited to deterministic decisioning. Placed inside a governed process, the same model becomes highly reliable at the narrower job it's actually good at.
"A better prompt alone guarantees identical results." Better prompting improves consistency. It doesn't replace business rules, formulas, and review controls — it just makes the inputs to those rules more reliable.
"We should make AI calculate everything." In a governed workflow, AI classifies and extracts; deterministic systems calculate and rank. Handing the final calculation to the model is exactly the step that reintroduces the variability the rest of the framework was built to remove.
"Human reviewers are always consistent." They aren't, when the rubric is unclear — human scoring drifts too. AI variability often just makes visible a governance gap in the scoring process that already existed before AI arrived.
Management Checklist
| Control area | Question to ask | Recommended action |
|---|---|---|
| Rubric | Have we defined what each score means? | Create score anchors for each dimension. |
| Prompt | Are all users using the same instruction? | Publish a controlled prompt template. |
| Calculation | Is the final priority calculated or inferred? | Use formula-based scoring for final ranking. |
| Evidence | Can we defend each classification? | Require source quotes or row references. |
| Audit | Can we reproduce the process later? | Record prompt, rubric, and workbook versions. |
| Review | What happens when the AI is uncertain? | Use confidence thresholds and human review. |
The Practical Operating Principle
AI-assisted analysis genuinely improves the speed and depth of business review, especially against large volumes of unstructured comments. The open question was never whether AI can analyse the data — it's whether the organisation has defined what consistent analysis means for that data.
A team that hands the model a broad prompt like "score and prioritise this feedback" is delegating business judgement to a probabilistic system, and then acting surprised when it behaves like one. A team that hands it an explicit rubric, a standard prompt, and a deterministic formula has converted the same model from an uncontrolled judge into a governed analytical assistant — without losing any of the speed that made AI worth adopting in the first place.
Use AI to interpret information. Use business rules to make the decision consistent.
Further reading: Berk Atil et al., "Non-Determinism of Deterministic LLM Settings", arXiv, 2025; Jiayi Yuan et al., "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference", arXiv, 2025; Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference", 2025.