The Engineering Debt Behind AI-Built Healthcare MVPs

Five evidence gates can help leaders decide what to retain, what to redesign and what must wait before real patient information enters a product.

By Silvio Sampaio

A healthcare prototype can demonstrate a compelling workflow without demonstrating that the organization can operate it responsibly. Before release, leaders need answers to less visible questions: who can retrieve a record, what happens when an integration fails, and can the team recover after an outage?

The proposed assessment uses five gates: data boundaries, authorization, clinical integration, recovery and runtime AI. It is a release framework, not a compliance certification or a claim that AI-generated software is inherently unsafe.

Separate two AI questions

AI-assisted development and runtime AI create different review scopes. For generated application code, assess architecture, dependencies, tests and maintainability. For an application that also calls a model in production, assess the information it receives and the actions its output can influence.

Do not allow a successful demonstration in one scope to stand in for evidence in the other. Assign owners and acceptance criteria separately.

Gate one: account for data

Map sensitive information across databases, storage, logs, analytics, support tools, backups and model-provider connections. Record each destination's purpose, access model, retention and responsible party.

HIPAA applicability depends on whether the organization is a covered entity or business associate, not simply whether an application concerns health. HHS makes that distinction in its guidance on covered entities and business associates.

HHS's HIPAA cloud-computing guidance requires applicable business associate agreements and an organization's own risk analysis. Selecting a cloud provider does not transfer every application responsibility to that provider.

The evidence gate is a reviewed data-flow inventory and responsibility allocation, including indirect destinations. Unknown destinations should block the affected data flow until assessed.

Gate two: prove authorization

A working login screen is not an access-control test. Attempt cross-organization access, patient substitution and unauthorized administrative operations through backend interfaces.

For integrated applications, distinguish negotiated API scopes from the underlying permissions that govern each request. HL7's SMART App Launch 2.2, "Scopes and Launch Context," explicitly preserves that distinction.

Require an access model supported by negative tests and a documented revocation process. Identify who approves privileged and support access; operational convenience should not become an undocumented exception.

Gate three: reconcile integration

One successful API response establishes connectivity, not reliable workflow completion.

HL7's FHIR Clinical Safety checklist covers resource lifecycle, patient links, error handling and validation. These concerns belong in integration acceptance tests, not merely implementation notes.

Exercise duplicate messages, delayed results, corrected records, partial responses and outages. Define the authoritative source for each data element and the handling of contradictions.

Require a reconciliation process with an accountable operator. If an event cannot be processed safely, expose the failure instead of silently discarding it or presenting an incomplete workflow as successful.

[FIGURE #01]

Figure 1. Proposed release gates for an AI-built healthcare MVP. Each gate requires evidence and an accountable owner. This framework is not a compliance certification.

Gate four: demonstrate recovery

HHS's Security Rule summary identifies risk analysis, audit controls, incident procedures and contingency planning among regulated entities' responsibilities.

For release assessment, require a witnessed restore exercise in a protected environment. Verify that the application works with the recovered state and record unresolved failures.

An enabled backup feature is not recovery evidence. Likewise, a healthy server does not prove that an integration or patient workflow works.

Set recovery objectives appropriate to the workflow, identify alert owners and document rollback and communication responsibilities.

Gate five: bound runtime AI

If the product calls a model in operation, define what it may draft, retrieve and execute. Keep authorization outside the model and require action-specific review where appropriate.

Use task-specific evaluation cases covering unsupported claims, missing evidence and inappropriate actions. NIST AI 600-1, the Generative AI Profile, provides a risk-management reference; it does not certify clinical fitness.

Place prompts, models and retrieval settings under change control. Define conditions for pausing a feature and evidence for resuming it. If the production product has no runtime AI, document that scope rather than inventing an unnecessary model-governance program.

Rebuild according to evidence

Avoid both automatic approval and an automatic rewrite. Retain useful components when their behavior is understandable, testable and supportable.

Before release, each owner should identify the evidence reviewed, remaining limitations and conditions that would reverse the decision. Unresolved issues may justify narrowing the release rather than treating readiness as an all-or-nothing label.

The value of a prototype is early learning. The production transition should preserve that learning while replacing assumptions with inspectable engineering decisions.