Holden Jolley
Azure Solution Architect
All projectsResumeGitHubLinkedInContact
RELIABILITY · DISTRIBUTED SYSTEMS

Distributed E-Commerce Platform

A production-oriented Azure reference architecture for a commerce backend that balances responsive checkout, inventory consistency, asynchronous processing, private connectivity, observability, and regional resilience without defaulting to Kubernetes complexity.

Dev / Test / Prod Bicep validatedARM What-If validated
Architecture at a glanceAzure preflight green
Front Door
+ WAF
→App Service
API + workers
→Cosmos DB
+ Service Bus
Availability2-region compute reference
ConsistencySingle inventory authority
OperationsApp Insights + Log Analytics
ProblemKeep checkout responsive without making inventory consistency ambiguous.
DecisionManaged regional App Service plus Cosmos DB and Service Bus behind Front Door.
Key tradeoffAPI compute is multi-region; Service Bus continuity remains an explicit follow-on design problem.
REQUIREMENTS & CONSTRAINTS

Keep checkout responsive without making consistency vague.

The customer-facing API should acknowledge work quickly while downstream inventory, payment, shipping, and analytics can proceed independently. The design therefore keeps the synchronous path small, makes inventory reservation an explicit consistency boundary, and pushes long-running work into asynchronous consumers.

The reference must also be reproducible from source control, use managed identity by default, keep data-plane services on private paths where practical, and demonstrate regional resilience without pretending every stateful dependency is automatically active-active.

Customer pathShort synchronous checkout path
ConsistencyExplicit inventory authority
SecurityManaged identity + private endpoints
OperationsObservable failure paths + runbooks
ARCHITECTURE CONSTRAINTS

Requirements decide where complexity belongs.

No Kubernetes by defaultRegional HTTP APIs and workers need managed compute, not necessarily a Kubernetes control plane. App Service reduces platform overhead while retaining identity, VNet integration, health checks, slots, and scale options.
Async after acceptanceInventory, payment, shipping, and analytics work is decoupled through Service Bus where the business flow does not require the customer request to wait.
Separate operational and analytical pathsCosmos DB holds commerce state while business events can be projected into Data Lake without placing analytics on the checkout critical path.
Dependency-specific resilienceFront Door, App Service, Cosmos DB, and Service Bus have different recovery characteristics; RTO and RPO cannot be reduced to one generic “multi-region” claim.
REQUEST & EVENT FLOW

One platform, intentionally separate failure domains.

Customers
→
Front Door
+ WAF
→
Regional API
App Service
→
Cosmos DB
+ Service Bus
Azure Front DoorWAFApp ServiceCosmos DBService BusKey VaultData Lake StorageVNetsPrivate EndpointsPrivate DNSApplication InsightsLog Analytics
QUALITY ATTRIBUTES

The architecture is tied to non-functional requirements, not a service checklist.

Availability

Front Door and a secondary App Service origin are the production-reference response to regional API failure. Runtime failover behavior still needs to be exercised after deployment.

Consistency

Inventory reservation and a single authoritative Cosmos write region keep stock ownership explicit instead of treating active-active writes as a free performance feature.

Messaging continuity

The current single Service Bus namespace is called out as a real regional dependency. The design does not claim an async-processing RTO until continuity is deliberately implemented and tested.

Security

Front Door/WAF is the intended ingress, data services move to private paths in the production profile, and managed identities replace reusable application secrets.

Observability

Application Insights and Log Analytics are part of the platform because retries, dead letters, dependency failures, and regional recovery need operational evidence.

Cost & maintainability

Development stays lower-cost, production adds resilience only where requirements justify it, and App Service avoids Kubernetes operational overhead until Kubernetes-specific needs appear.

COMPONENT RESPONSIBILITIES

The architecture makes ownership explicit.

API

Owns product, cart, checkout acceptance, and order retrieval. It stays focused on the customer-facing request path rather than performing every downstream side effect inline.

Inventory worker

Owns reservation, release, and commit transitions. Reservation is the deliberate consistency boundary used to prevent ambiguous stock ownership.

Payment worker

Owns payment-provider interaction and requires idempotent behavior so retries do not duplicate external side effects.

Shipping worker

Handles fulfillment transitions independently from the checkout request so shipping failures do not force the customer-facing API to remain open.

Analytics worker

Copies business events into Data Lake. Analytics can lag or recover independently because it is intentionally outside the checkout critical path.

Platform services

Cosmos DB holds operational state, Service Bus carries asynchronous work, Key Vault protects secrets/keys, and Application Insights + Log Analytics provide the operational evidence needed to diagnose failures.

REGIONAL DESIGN

Multi-region compute does not magically make every dependency multi-region.

The production reference places API/worker compute in two regions behind Front Door. Cosmos DB is modeled across both regions with one authoritative write region and automatic failover. Front Door can route around an unhealthy API origin and Cosmos can promote the configured secondary region.

Service Bus continuity is intentionally treated as a separate architecture problem. A regional Service Bus outage can pause asynchronous work in the current baseline, so the project does not claim a messaging RTO that has not been designed and tested.

API failoverFront Door origin health
CosmosSingle write authority + failover
MessagingRegional continuity backlog
AnalyticsHigher recovery tolerance
FAILURE TOPOLOGY

Different failures take different recovery paths.

Primary API
unhealthy
→
Front Door
health routing
→
Secondary API
origin
→
Customer traffic
continues
Cosmos write
region unavailable
→
Automatic
failover
→
Secondary region
promoted
→
Observed RTO/RPO
must be tested
Service Bus
regional outage
→
Single namespace
dependency
→
Async processing
pauses
→
Continuity design
required
FAILURE BEHAVIOR

Design for what breaks, not only for the happy path.

Primary API origin unhealthy

Front Door routes to the healthy secondary origin in the production reference profile.

Worker crashes mid-message

The locked message becomes available for redelivery. Consumers must be idempotent because the crash may happen after an external side effect but before message completion.

Poison message

Retries stop at the configured limit and the message moves to the dead-letter queue for operational handling rather than retrying forever.

Payment provider unavailable

Retry/backoff applies, the order remains incomplete, and reserved inventory must eventually be released instead of becoming permanently stranded.

Cosmos write region unavailable

Cosmos failover promotes the configured secondary region. Observed recovery still needs to be validated under a deployed test.

Service Bus regional outage

Asynchronous processing can pause. The current reference records that limitation instead of hiding it behind a generic multi-region label.

POST-DEPLOYMENT RESILIENCE TESTS

What I would prove at runtime.

01Traffic and regional failover

Make the primary API origin unhealthy and verify Front Door moves traffic. Exercise controlled Cosmos regional failover and record observed recovery instead of assuming the service behavior meets the target.

02Message integrity

Crash a worker after a side effect but before completion, prove idempotent redelivery, inject a poison event, and verify dead-letter handling and the associated runbook.

03Business consistency

Simulate payment timeouts and verify inventory release, then run a high-contention inventory race to prove the reservation boundary prevents overselling.

KEY DECISIONS

Complexity is introduced only where the requirement earns it.

App Service instead of AKS

The workload needs independently deployable API and worker processes, not a Kubernetes control plane. App Service keeps the operational surface smaller while preserving scale, managed identity, VNet integration, health checks, and deployment slots.

Service Bus for asynchronous commerce work

Payment, shipping, inventory, and analytics can fail or retry independently without forcing the customer request to wait for every downstream dependency.

Cosmos DB with a clear inventory authority

The design prioritizes a defined inventory-consistency boundary rather than enabling multi-master writes everywhere and creating ambiguous stock ownership.

Private networking for data services

Private endpoints and DNS keep data-plane services off unnecessary public paths while managed identities remove application secrets from service-to-service authentication.

EVIDENCE LEVELS

What has been proven—and what has not.

Design evidence

Requirements, constraints, NFRs, ADRs, diagrams, security, reliability, operations, and known limitations are documented in source control.

Compiler evidence

Dev, test, and production Bicep and parameter validation is green.

Azure control-plane evidence

GitHub Actions authenticates with OIDC and the dev architecture completes live subscription-scope ARM What-If evaluation.

Runtime evidence

Not claimed yet. Application binaries, real failover, performance, recovery, and end-to-end business behavior still require deployment and testing.

Cloud validated
Compiler and live Azure control-plane validation are complete for the dev architecture.

GitHub Actions lints and builds the Bicep template and all dev, test, and production parameter files, authenticates to Azure through secretless OIDC, and runs a subscription-scope Resource Manager What-If. This proves the infrastructure model can be evaluated by Azure; it does not substitute for runtime resiliency, load, integration, or recovery testing. The deployment job remains gated and has not been run.

Current boundary: this is a reference architecture package containing the architecture, Infrastructure as Code, contracts, deployment tooling, ADRs, testing guidance, and operational documentation. Application binaries, end-to-end runtime evidence, and a tested Service Bus regional-continuity design remain outside the current validated boundary.