Can a specialty pharmacy use LLMs with PHI?

What HIPAA actually requires

HIPAA is technology-neutral. It does not name LLMs, and it does not need to. The same rules that govern your pharmacy management system and your EHR integrations govern an AI vendor:

• A business associate agreement (BAA). Any vendor that creates, receives, maintains, or transmits PHI on your behalf is a business associate. No signed BAA, no PHI — full stop. The major enterprise LLM platforms will sign one; consumer AI products will not.
• The minimum necessary standard. Send the model only the PHI the task requires. A refill-status question does not need a full medication history attached.
• Safeguards. Access controls, encryption in transit and at rest, and workforce policies — the Security Rule applies to AI workflows exactly as it applies to everything else that touches PHI.
• Accountability. If an auditor asks how AI touches PHI in your operation, “we’re not sure” is the most expensive possible answer. You need to be able to produce the record.

None of this is exotic. It is the same discipline you already apply to your EHR, your fax vendor, and your billing clearinghouse — applied to a new category of vendor.

The reference architecture

The design principle: PHI never leaves your compliance boundary — instead, the BAA extends your boundary to cover the vendor.

Three zones, one rule.

Zone 1 — Your systems. The pharmacy management system, the EHR or Epic connection over HL7/FHIR, the intake queue (fax, portal, referrals), and the staff who work them. Nothing here changes.

Zone 2 — The AI gateway. The part most organizations skip, and the part that makes the whole thing defensible. Every AI request from your staff or systems passes through a control layer you own:

• Identity first. Single sign-on and role-based access. No anonymous prompts, ever.
• Minimum necessary, enforced. The gateway strips or masks PHI the task doesn’t require before anything leaves your systems.
• Approved use cases only. The gateway permits the workflows your policy has sanctioned and blocks everything else. A policy that lives in a PDF is a suggestion; a policy that lives in the gateway is a control.
• Audit logging. Every prompt and every response, recorded, attributable to a person, retained on your terms. This log is what turns “we use AI” into “we can prove how we use AI.”

Zone 3 — The LLM provider, under BAA. An enterprise API endpoint — not a consumer chat app — with four contractual and technical properties: a signed BAA, zero data retention, no training on your data, and encryption in transit and at rest. The BAA is what stretches your compliance boundary around the vendor. Without it, sending PHI to a model is an impermissible disclosure regardless of how good the security is.

The return path matters as much as the outbound one. Model output routes back through the gateway, gets logged, and lands in front of a pharmacist or technician for review before anything touches a patient record or a payer submission. Human review is not a courtesy — for clinical and coverage-affecting workflows, it is the control your auditors and your conscience both require.

And the blocked lane. Consumer AI tools — personal ChatGPT accounts, free-tier assistants, browser extensions — sit outside every boundary in the diagram: no BAA, no audit trail, no recall. They are blocked by policy and, wherever possible, by network controls. A ban without a sanctioned alternative pushes usage underground; the gateway is the sanctioned alternative.

The five conditions, as a checklist

Meet all five and the question changes from “can we use LLMs with PHI” to “which workflows first.” Miss any one and you have built a breach with extra steps.

Where specialty pharmacies get this wrong

• The silent default. Staff are already pasting referral details into consumer tools because it saves them twenty minutes. Surveys consistently put unsanctioned AI use at a majority of employees. The exposure exists today; the only question is whether you can see it.
• BAA-by-assumption. “The vendor is big, surely it’s covered” is not a compliance posture. Enterprise tiers and consumer tiers of the same product carry different terms. Verify the tier you are actually on.
• Policy without architecture. An approved AI policy with no technical enforcement is shelfware. Auditors have learned to ask for the logs.
• Automation without review. Letting model output flow unreviewed into prior auth submissions or patient communications trades administrative burden for clinical and regulatory risk. The time savings survive human review; the risk does not.

What this makes possible

With the architecture in place, the workflows with the strongest ROI in specialty pharmacy are the ones drowning your staff today: prior authorization drafting, referral intake summarization, benefits investigation notes, refill-queue triage, and payer correspondence — each measured in staff hours recovered per week, each with a pharmacist in the loop.