ChatGPTCast journal
Compare two prompts by changing one instruction
When two prompts differ in many ways, a better-looking response is difficult to explain. Plan a comparison that changes one instruction at a time. Use a harmless invented task and a written scoring rule, keeping the result limited to the examples you actually inspect.
By ChatGPTCast editorial · · 2 min read
Choose one task and one intended change
Invent several short notices that need to be turned into a two-column summary. Use fictional events and no private information. Write one prompt, then make a second version that changes only the instruction you want to examine, such as how unknown fields should be represented. Keep the source notices the same for both versions. Record the exact instructions before running anything. Do not edit an input after seeing a poor answer and still call the pair a comparison of one change; that would introduce another explanation for the difference.
Score against a rule written in advance
Define the expected fields and mark which values are present, absent or ambiguous in each invented notice. Decide how omissions and invented details will be counted. When you inspect the outputs, use those criteria rather than choosing the response that sounds more polished. If possible, assess the outputs without the prompt labels visible to reduce your own preference for the revision. Keep every observed response and note the model or product context available to you, without inventing a version or describing a manual check that never happened.
Report the difference without a universal claim
Write which examples improved, which worsened and which were unchanged under your rule. A small comparison cannot establish broad accuracy or reliability across new tasks, languages and settings. Do not give the demonstration tools that can send, buy or publish anything; the exercise needs text outputs only. End with one useful next test rather than a claim that the improved prompt is safe for every workflow. A transparent comparison helps viewers understand a particular instruction choice while preserving uncertainty about behaviour beyond the test.
- Change one instruction while keeping test inputs fixed.
- Write expected results and scoring rules first.
- Retain unfavourable outputs as part of the comparison.
- Report observed differences without promising general reliability.