Bulk Data Export: Wizard, Jobs, and FHIR Standards

By David Le -- Part 8 of the FhirHub Series

Healthcare organizations need to move data in bulk -- for analytics, regulatory reporting, data migration, and research. FhirHub implements a bulk data export system with a guided 4-step wizard, asynchronous job processing, and support for NDJSON, JSON Bundle, and CSV formats.

The Export Wizard

The export wizard walks users through configuration in four steps:

1. Select Resources -- Choose which FHIR resource types to export
2. Date Range -- Optionally filter by time period
3. Options -- Select format and configure references
4. Review -- Confirm settings before starting

// frontend/src/components/export/export-wizard.tsx
const steps = [
  { id: 1, name: "Select Resources", description: "Choose FHIR resource types" },
  { id: 2, name: "Date Range", description: "Filter by time period" },
  { id: 3, name: "Options", description: "Configure export settings" },
  { id: 4, name: "Review", description: "Confirm and start export" },
];

Step 1: Resource Type Selection

Users choose from 10 FHIR resource types, each with a description and record count:

const resourceTypes = [
  { id: "Patient",           label: "Patients",           description: "Patient demographics and identifiers" },
  { id: "Observation",       label: "Observations",       description: "Vital signs, lab results, assessments" },
  { id: "Condition",         label: "Conditions",         description: "Diagnoses and health conditions" },
  { id: "MedicationRequest", label: "Medications",        description: "Medication orders and prescriptions" },
  { id: "DiagnosticReport",  label: "Diagnostic Reports", description: "Lab reports and imaging results" },
  { id: "Encounter",         label: "Encounters",         description: "Patient visits and admissions" },
  { id: "Procedure",         label: "Procedures",         description: "Clinical procedures performed" },
  { id: "Immunization",      label: "Immunizations",      description: "Vaccination records" },
  { id: "AllergyIntolerance",label: "Allergies",          description: "Allergy and intolerance records" },
  { id: "DocumentReference", label: "Documents",          description: "Clinical documents and attachments" },
];

A "Select All" / "Clear" toggle makes it easy to choose everything or start fresh.

Step 3: Export Formats

Three formats are supported:

NDJSON is recommended and selected by default -- it's the format specified by the FHIR Bulk Data Access specification and supports streaming for large datasets.

Export Configuration

The wizard produces a typed configuration object:

export interface ExportConfig {
  resourceTypes: string[];
  dateRange: { start: string; end: string } | null;
  format: "ndjson" | "json" | "csv";
  includeReferences: boolean;
}

The includeReferences option automatically includes resources referenced by the selected types. For example, exporting Encounters will also include referenced Practitioner resources.

Job Lifecycle

Export jobs follow an asynchronous lifecycle:

pending → in-progress → completed
                      → failed → (retry) → pending
                      → cancelled

The job DTO captures the full state:

export interface ExportJobDTO {
  id: string;
  status: ExportStatus;  // "pending" | "in-progress" | "completed" | "failed" | "cancelled"
  resourceTypes: string[];
  format: ExportFormat;
  createdAt: string;
  completedAt?: string;
  expiresAt?: string;
  progress?: number;
  fileSize?: number;
  downloadUrl?: string;
  error?: string;
  dateRange?: { start: string; end: string };
  includeReferences?: boolean;
}

Key fields:

• progress -- Percentage complete (0-100) during in-progress
• downloadUrl -- Available after completed
• expiresAt -- Completed exports auto-expire for storage management
• error -- Error message when failed

API Endpoints

The ExportsController provides full CRUD plus lifecycle management:

[ApiController]
[Route("api/exports")]
[Authorize]
public class ExportsController : ControllerBase
{
    [HttpGet]                // List all export jobs
    [HttpGet("{id}")]        // Get specific job
    [HttpPost]               // Create new export job
    [HttpPost("{id}/cancel")] // Cancel a running job
    [HttpPost("{id}/retry")]  // Retry a failed job
    [HttpDelete("{id}")]     // Delete a job (admin only)
}

Authorization is enforced at two levels:

• CanManageExports -- Required for listing, creating, cancelling, and retrying (admin, practitioner)
• CanDeleteExports -- Required for permanent deletion (admin only)

All write operations are rate-limited with the WriteOperations sliding window limiter.

The Export Job List

The ExportJobList component (export-job-list.tsx) displays all export jobs with their current status, progress, and available actions:

Each job row shows:

• Status badge (color-coded)
• Resource types included
• Format
• Creation date
• Progress bar (for in-progress jobs)
• Download button (for completed jobs)
• Cancel/Retry/Delete actions

Design Decisions

Why a wizard instead of a form? Bulk exports are consequential operations that can take significant time and server resources. The 4-step wizard forces users to think through their choices and review before committing. The review step shows an estimated file size and duration.

Why asynchronous? Healthcare datasets can be large. A 10,000-patient export with full observation history could produce gigabytes of data. Async jobs let the server process in the background while the user continues working.

Why separate cancel, retry, and delete? These are distinct operations:

• Cancel stops a running job (preserves the record)
• Retry creates a new attempt from a failed job's configuration
• Delete permanently removes the job record (admin only, irreversible)

What's Next

In Part 9, we'll examine the .NET 8 API gateway's middleware pipeline in detail -- exception handling, security headers, rate limiting, and structured logging.

---

Find the source code on GitHub Connect on LinkedIn