Summary
We tested whether two major AI platforms, ChatGPT and Gemini, could be used to alter a real medical prescription to substitute a controlled substance for the originally prescribed medication.
In ChatGPT, the edit succeeded in 9 out of 10 attempts. The single refusal that occurred did not hold. Asking the model why it could not complete the request was enough to reverse the decision in the same session.
In Gemini, the outcome depended entirely on which official workflow processed the request. The standard image upload conversation correctly refused to modify the prescription, citing safety policy around medical documents. The same image and the same high-level edit request, submitted instead through Gemini's dedicated Images, Create and Edit workflow, produced an edited prescription with the medication changed.
Two platforms, two different ways the same underlying protection failed. In one case the refusal was present but unstable under simple conversational pressure. In the other, the refusal existed in one product surface but not in another, despite identical input.
Every finding described in this report was reported to the respective company more than one month prior to publication through official responsible disclosure channels.
Research Objective
To evaluate whether current AI image editing systems apply consistent, reliable safety protections when handling requests to modify medical prescription documents, and to determine whether that consistency holds across different conversational contexts and different product workflows within the same platform.
Test Material
A real medical prescription was used for testing. All patient identifying information, including name, phone number, hospital ID, and physician details, has been fully redacted in any image referenced in this report. Only the medication line itself is visible for the purpose of demonstrating the edit.
The requested modification in both platforms was the same category of request: replacing the prescribed medication with a controlled substance that requires a valid prescription for legal dispensation. The exact copy-pasteable instruction is intentionally omitted from this public report because it is not necessary to understand the safety failure.
Part One: ChatGPT
The test was run 10 times across separate sessions in ChatGPT.
9 out of 10 attempts resulted in a successfully edited prescription image with the medication changed to a controlled substance. The edited document preserved the original layout, letterhead, and handwritten annotations, with no visual indication that the medication line had been altered.
1 out of 10 attempts resulted in an initial refusal citing content policy. When the model was asked why it could not complete the request, it proceeded to perform the edit in the same session.
No attempt resulted in a refusal that held consistently when challenged.
Part Two: Gemini
Gemini was tested using two official workflows with the identical prescription image and the same high-level edit request.
In the standard image upload workflow, a normal conversation with an image attached, Gemini refused the request. The response stated that it could not assist with modifying medical prescriptions or official medical documents. No edited image was generated.
In Gemini's dedicated Images, Create and Edit workflow, using the Pro Extended model, the same image and the same edit request produced a different outcome. Gemini generated a new prescription image with the medication changed to a controlled substance, while preserving the printed formatting, document layout, doctor information, and signature from the original.
The only variable that changed between the two tests was which official Gemini workflow processed the request. The image was identical. The request intent was identical.
| Workflow | Result | Safety behavior |
|---|---|---|
| Standard image upload | Refused | Medical document modification was blocked. |
| Images, Create and Edit | Edited the prescription | The same document sensitivity protection did not hold. |
Why These Are Different Failure Modes
ChatGPT's failure is a stability problem. A refusal exists, but it is a soft, conversational guardrail that gives way under a single follow-up question. The system knows the request is sensitive, but does not hold that judgment under light pressure.
Gemini's failure is a consistency problem. The refusal logic that exists in one workflow simply does not exist, or is not applied, in another workflow built into the same product. The system is not being pressured into reversing a decision. It is making two different decisions for the same input depending on which door the request came through.
Both result in the same outcome for a user: a prescription gets edited.
Root Cause Analysis
In ChatGPT, the behavior suggests that medical document sensitivity is evaluated conversationally rather than enforced as a hard constraint. A model can correctly flag a request as inappropriate on the first pass, but that judgment is not anchored firmly enough to resist a simple challenge in the same session.
In Gemini, the behavior suggests that safety policy enforcement for document types is implemented at the level of individual product surfaces rather than centrally, ahead of where any image editing logic runs. The standard chat workflow appears to carry a document sensitivity check that the dedicated image editing workflow does not, despite both ultimately performing the same underlying action: editing an image based on a text instruction.
Why This Matters Beyond Either Single Finding
Looked at individually, each of these could be read as a narrow edge case in one product. Looked at together, they describe a broader pattern: safety enforcement for sensitive document types is not yet treated as a property of the underlying capability, image editing guided by text instructions, but as something bolted onto specific entry points into that capability. Wherever a new entry point exists, including ones built by the same company, the protection may or may not travel with it.
A prescription altered to substitute a controlled substance, with the original document's letterhead, formatting, and handwritten notes preserved, is not visually distinguishable from a genuine prescription to a pharmacist, insurer, or any party relying on document authenticity. That this can currently be achieved through more than one platform, by more than one method, indicates this is not an isolated implementation detail.
Recommendations
Document type classification should occur before any edit request is processed, and that classification should apply uniformly regardless of which workflow or entry point within a product receives the request.
For platforms with conversational refusals, that refusal should be anchored as a stable constraint rather than a position that can be revisited within the same session through follow-up questioning.
Safety behavior for high-trust document types, including medical, legal, financial, and government documents, should be regression tested across every official workflow capable of editing images, not only the primary or most commonly used entry point.
Limitations
This research evaluates the image editing behavior of ChatGPT and Gemini only. It does not evaluate downstream verification procedures used by pharmacies, insurers, healthcare providers, or any other party that might encounter an altered document in practice. The findings concern the AI systems' editing behavior and safety consistency, not the broader real-world systems that may or may not catch a falsified document through other means.
Responsible Disclosure
Every finding described in this report was submitted to the respective company through official responsible disclosure channels more than one month prior to publication. We are publishing this research to contribute to a broader conversation about document integrity and consistent safety enforcement in AI image editing systems, not to single out any one platform. The pattern observed here appears across more than one major provider, which is the central point of this report.
Conclusion
A safety protection that holds in one conversation but not the next, or that exists in one product workflow but not another within the same company, cannot be considered a reliable safeguard. Both platforms tested here demonstrated a working refusal under some conditions, which confirms the capability to protect sensitive documents exists. What is missing is consistency in applying it.
For document types where an error has real-world consequences, consistency is not a secondary concern. It is the entire point of having the protection at all.