Diagnose a stalled AI pilot from its run notes
The pilot has gone quiet. Before calling it a training problem, read what happened on the last few attempts.
What you'll make
a short diagnosis tied to pilot notes, with one small follow-up test.
What you'll need
ChatGPT in your work account, a few notes about attempts and one person who can check them. Allow 15 minutes. Use process notes; leave out personal assessments and private conversations.
Example result — illustrative diagnosis:
| Finding | What supports it |
|---|---|
| Source access failed | Attempts 2 and 3 couldn’t open the route file |
| Motivation is unknown | No note records why people stopped |
| Next test | Retry with the current route file attached |
1. Assemble the attempt notes
Open ChatGPT, select Chat › New chat. Paste a small set of dated run notes, stating what was attempted, what happened and what was missing. Don’t turn an absent observation into a zero or a reason.
Read these pilot notes and make a diagnosis table. Give each finding, the attempt numbers supporting it and anything we still need to know. Separate observed obstacles from possible explanations. Do not infer people's motivation.
Museum visitor-route update pilot:
1. Current route file attached. Draft checked and corrected one room name.
2. Route file link couldn't be opened. No draft completed.
3. Same link couldn't be opened. No draft completed.
4. No run note recorded.Why this works: attempt numbers make the diagnosis traceable. Separating an observation from an explanation avoids a confident story about an incomplete log.
What you should see: two access failures, one checked draft and one missing record.
2. Check the proposed explanation
Compare every finding with the notes. An access error supports an access problem. It doesn’t establish poor training, lack of interest or whether the output would have been useful.
Why this works: fixing the wrong cause can turn a small technical obstacle into a larger programme.
What you should see: findings you can point to, with unresolved questions still visible.
3. Design one test of the obstacle
From the checked diagnosis, propose one small test of the repeated obstacle. State the input to supply, what to try, what to record and the result that would support or weaken the explanation. Keep it to one attempt; do not recommend a wider rollout.Why this works: a test asks whether the explanation survives another attempt. It doesn’t reward the most persuasive diagnosis.
What you should see: a retry with the current route file attached, recording access, draft quality and corrections separately.
Check the result
- Does every observed obstacle point to a run note?
- Is attempt 4 marked missing rather than failed?
- Could the next test disprove the leading explanation?
The usual failure: the answer diagnoses resistance from silence.
Remove the claim that the team resisted the pilot. The notes contain no evidence about motivation. Mark attempt 4 as missing, keep the two access failures, and narrow the next test to whether attaching the current file removes that obstacle.Worked example
Fictional example.
Attempt 1: current museum route notes attached; draft completed.
Attempt 2: source file could not be opened.
Attempt 3: source file could not be opened.
Attempt 4: outcome not recorded.The museum log contains one completed draft, two inaccessible-file attempts and one blank entry. The weak diagnosis says: “Adoption is low because staff lack confidence.”
After the repair, it says: “The recorded repeated obstacle is source access. Motivation and the outcome of attempt 4 are unknown.” The next attempt uses an attached current file. If access works but the output still needs extensive repair, the next investigation changes.
Missing evidence stays missing. That makes the test useful even if it doesn’t confirm the first explanation.
Go deeper
Keep findings separate from fixes
An explanation can be correct while the proposed fix is unhelpful. Add a separate line for the change you tried and what happened afterwards. A new training session isn’t evidence that training was the obstacle.