Twelve engagements. The numbers, not the narrative.

What was breaking. Three problems that looked separate and were not. Rating ran through integrations with no isolation, so a single carrier’s change took the whole flow down. Tracking had never been unified — clients did not trust the visibility they were given, support drowned in “where is my shipment” tickets, and account managers pulled statuses by hand out of separate carrier portals. Freight audit happened retroactively in Excel: fuel surcharge, detention and residential reconciled after the fact, with some delivered services never billed at all. Combined monthly write-offs and unbilled charges came to $45,000.

What we built. The engagement began with a CTO-sponsored technical audit that sequenced the work by revenue impact rather than technical tidiness.

Carriers went behind one internal API with a circuit breaker per carrier, so one failing carrier no longer cascades and adding a carrier became configuration rather than code. Tracking consolidated into a canonical event model behind a single client-facing interface. Freight audit moved into a Spring Boot service consuming invoices over Kafka, comparing billed amounts against contracted rates and accessorial rules automatically.

Then we closed the revenue side: every tariff-bearing operational event now creates a billable record by construction. A delivered service cannot quietly go unbilled.

The real constraint. They could not stop shipping while we worked, which ruled out the clean version of this project — freeze, rebuild, cut over. So sequencing became the design decision. Rating isolation went first because an outage was costing money that week, and known-bad code elsewhere was deliberately left alone for months while it was not the expensive thing. Revenue impact decided the order, not architectural conscience.

What changed.

What was breaking. The leakage was structural rather than accidental. Invoices arrived from eight carriers in incompatible formats and an accounts payable team of four normalized them by hand, which had stopped scaling against 30% annual volume growth. There was no drill-down from an invoice line to the operational event behind it, so every client question turned into a manual investigation. And nothing about the disputes themselves was measured — win rate by carrier, monthly recovery, dispute aging. Disputes ran blind, which meant nobody could tell a good carrier relationship from an expensive one, and every renewal was negotiated on memory.

What we built. An automated freight audit pipeline. PDF ingestion with per-carrier extraction rules, EDI 210 parsing for the structured feeds, everything normalized into one canonical invoice object.

The audit engine checks every line against the contracted rate card — base rate, fuel surcharge formula, the full accessorial schedule. That let the dispute threshold drop from $50 to $5, with each flagged variance auto-assembling its own evidence package: contracted rate, billed amount, variance, and the contract clause it violates. Every invoice line traces back to its shipment event. The dispute module generates a formatted letter in one click, and recovery is tracked live by carrier, aging and status — which turned the renewal conversation from an anecdote into a number the broker owns.

The real constraint. Dropping the threshold tenfold only works if assembling evidence is free. That inverted the project: the hard part was never the audit logic, it was extraction that stays correct when a carrier silently changes its PDF layout. We used explicit per-carrier extraction rules and made extraction failures loud instead of letting them degrade into false variances.

What changed.

What was breaking. Shipments went out without a valid ASN and chargebacks accumulated before the account had turned profitable. The gap between the physical pallet and the transmitted 856 ASN ran about 4%, because the SSCC-18 GS1-128 label generator was a disconnected desktop tool and carton-pallet structure was typed into EDI by hand. Large retail chains issue compliance chargebacks at published rates against cost of goods, and they issue them regularly.

Nobody tracked 997 acknowledgments. Rejections accumulated in silence until a client called or a penalty letter arrived, which meant the first signal of an EDI failure was a commercial one.

What we built. An EDI accelerator with prebuilt transaction maps for large US retail-chain profiles, covering 850, 856, 810, 940 and 945, configured per partner without code changes.

We integrated the SSCC-18 label generator with warehouse pick confirmation, so confirming a carton simultaneously generates the GS1-128 label and the 856 ASN carton record. Physical and transmitted data now match by construction rather than by diligence — the discrepancy is not caught, it is made impossible.

We added 997 monitoring: every outbound transaction carries a response timeout, and rejections surface on the ops dashboard within 15 minutes with a notification to the account coordinator.

The real constraint. Prebuilt maps are a promise you cannot keep in a lab. A partner profile that works against documentation and fails against the retailer’s actual VAN is worse than no accelerator at all. So the accelerator was validated live: the client’s first new retail contract was connected through it during the project itself, on the client’s real deadline, before we had a second one to prove the pattern.

What changed.

What was breaking. Reliability was felt rather than measured. No SLOs on order ingest, carrier API or EDI flows. No runbooks, so the same failures were solved from scratch every time. Response times ran 45 to 90 minutes and the organization was paying its most expensive person for the absence of a process. A four-hour failure inside the peak window stops outbound entirely and costs $80,000 to $120,000 in SLA penalties.

Monitoring watched servers rather than the business. A dropped WMS integration or an interrupted EDI feed was discovered two to four hours later, when orders stopped moving and somebody noticed.

What we built. OpenTelemetry across all services, with business signals on one dashboard: order throughput, carrier API latency, EDI 945 success rate, scan event frequency, shipment creation rate.

An SLO per critical integration — warehouse order sync, carrier rate APIs, EDI 945 — with error budgets reviewed monthly with the client. Tiered on-call across three time zones: P1 alerts routed and escalated within 2 minutes, P2 within 10; nights and weekends fully owned by our SRE team. Forty-plus runbooks, monthly chaos engineering, and a service manager running escalation and quarterly SLO reviews.

The real constraint. The bottleneck was a person, and people are harder to remove from a process than code is. Taking night incidents away from the CTO meant asking the business to accept published numbers it could be held to — an SLO is a commitment in both directions, and the first month of error budgets is where that gets tested. The engineering was ordinary. Getting the organization to agree to be measured was the project.

What changed.

What was breaking. The client had changed vendors several times in two years, and every new team needed another two to three months to learn a C# TMS with carrier-rating logic, EDI flows and freight audit before it could contribute. Hiring was not closing the gap either: a senior C# engineer with logistics experience took four to six months to find, and most candidates knew .NET but had never seen a TMS, a WMS or an EDI feed.

What we did. Our tech lead screened specifically for C# engineers who had worked on logistics platforms, carrier APIs or EDI. Three were selected and joined the client's engineering organization — its backlog, its ceremonies, its day-to-day management. The tech lead ran weekly code review, set architectural standards and structured the handover of codebase context. Over six months the team took ownership of carrier integrations, billing logic and the EDI pipeline.

What changed.

What was breaking. At peak, that degradation meant queue buildup that puts cutoffs at risk — which in this business is not a performance metric, it is a shipment that does not go out. Nightly batch jobs lagged the inventory picture, so replenishment and routing decisions were being made on yesterday’s data. And there was no client-ready data layer at all, so every status or stock question from the operator’s own clients became a support ticket and a manual export.

What we built. A two-tier architecture: an operational tier for writes and current reads, an analytical tier for reporting and history. A bigger box would have bought perhaps a year — the contention was between two workloads with opposite access patterns sharing one table, and no amount of hardware settles that argument.

We partitioned the orders table by month with pg_partman and migrated all 3.2 billion rows incrementally and in parallel with no downtime. VACUUM now runs per partition in 15–20 minutes with far less impact on production. A read replica pool with streaming replication and PgBouncer separated reads from writes, and CDC via Debezium replicates changes into ClickHouse in near real time.

On top of ClickHouse we built a client-ready data layer with dbt, so the operator’s own clients get self-service access to their orders, stock and SLA metrics instead of asking a human.

The real constraint. There was no maintenance window and there was never going to be one — the order stream does not pause for a migration. That removed every textbook option. Rows had to move underneath live writes, in parallel, without ever leaving the table in a state the application could not read. The migration design was dictated entirely by the absence of permission to stop.

What changed.

What was breaking. Duplicate webhooks caused the same order to be processed twice. Static routing across three fulfillment centers produced systematic split shipments — double delivery cost, worse margin, and late deliveries whenever load was uneven. The product master had diverged across ERP, channels and warehouses, which surfaced as picking errors and wrong substitutions. Status notifications lagged 45 to 90 minutes, and missing-update complaints made up 18% of all support tickets. Across eight markets none of this stayed local: one channel’s duplicate webhook became a support queue in three countries, and the manual recovery ran on the same small ops team every time.

What we built. We replaced the middleware with an event-driven Go ingestion layer publishing to Kafka. Every event carries an idempotency key, so duplicate webhooks are filtered at entry and double processing is ruled out architecturally rather than patched after the fact.

A routing engine now evaluates real stock position, the delivery-cost matrix and the promised ship date to select a single fulfilling center — a split shipment became an exception that has to justify itself. Product-master alignment made the ERP the source of truth, with attribute changes replicated to channels and warehouses by event. The order state machine publishes every status transition, and notifications go out within seconds.

The real constraint. You cannot ask a marketplace to stop resending webhooks, and you cannot ask eight markets to align their channel behavior for your convenience. Everything upstream was non-negotiable, which meant correctness had to be enforced at our boundary or not at all. Idempotency at entry was not a refinement — it was the only place in the whole system where the problem could be solved once instead of eight times.

What changed.

What was breaking. Planners spent 70% of their time gathering data rather than deciding anything. Forty percent of the catalog had intermittent demand — months of zeros, then a single-buyer spike — which is exactly where standard forecasting models fail and exactly where being wrong costs the most. Lead times ranged from 3 to 45 days across twelve suppliers, but purchasing did not account for individual supplier reliability. And whatever the forecast produced was moved into the ERP by hand.

What we built. An ensemble rather than a single model: LightGBM with 40+ features for regular demand, roughly 60% of the catalog, and a Croston/neural-network hybrid for the intermittent 40%, calibrated to the industrial-components profile.

A supplier-reliability model built on 24 months of order history turns a per-supplier lead-time distribution into an automatic safety-stock adjustment, so an unreliable supplier costs inventory rather than service level.

Most importantly, the forecast went into the workflow instead of into a dashboard. Recommendations flow by API straight into the ERP replenishment module as draft purchase orders for one-click approval. Accuracy monitoring by segment triggers retraining when quality drops, so model drift is a handled condition rather than a future incident.

The real constraint. The model was never going to be the hard part. Six planners who had run this catalog for years were being asked to trust a system, and a forecast that arrives as a report gets overridden until it is ignored. So the deliverable was defined as a draft purchase order inside the tool they already used, with the planner keeping the final click. Adoption was designed before accuracy was optimized.

What changed.

What was breaking. Spend had no owner. LLM APIs were costing $42,500 a month, including calls running from staging into integrations nobody used. Two vendors had sold “AI features” that turned out to be regex underneath, never technically verified by anyone on the client side. Three genuinely promising ML projects were stuck in Jupyter notebooks with no deployment pipeline, no monitoring and no route to production — so the company was simultaneously overpaying for AI and getting none of it.

What we built. A structured audit first: every initiative got a status, a real ROI estimate and a technical-readiness rating, which made the portfolio legible to leadership for the first time.

We turned off the staging LLM API calls, which saved $14,000 in the first week. We dropped two pseudo-AI vendors after technical verification. Then we built an ML deployment pipeline — MLflow for tracking and versioning, FastAPI serving, Docker, CI/CD with quality gates — which gave the three stuck projects an actual route to production.

Then a governance rule that outlives us: no new AI initiative starts without defined success metrics, an ROI estimate and a named business owner. The first priority use case, automatic carrier invoice classification, went live under it.

The real constraint. The blocker was political, not technical. Somebody senior had approved the budget and two vendors, and the audit’s honest finding was that part of it bought nothing. That is the finding most consultants soften into a roadmap. We wrote it as a per-initiative rating with the evidence attached, so the conversation was about a document rather than about a person — and the governance rule at the end exists so the same decision cannot be made informally again.

What changed.

What was breaking. Nothing existed — no dispatch orchestration, no inventory reservation, no courier or picker apps. Beyond the blank page, two failure modes were guaranteed by the model itself. Concurrent orders on the same SKU at peak create oversell, and a customer who discovers an item is unavailable after paying is a refund, a support contact and usually a lost account — in a business that only works on repeat orders. And couriers work budget Android devices on unstable urban LTE, so a lost delivery event means a dispute with no evidence behind it.

What we built. The full platform in six months. Customer apps bind each address to the nearest hub and cap carts by weight and volume, so an undeliverable order cannot be placed. Pessimistic database transaction locking rules out overselling structurally rather than probabilistically.

Substitution logic in the picker flow offers an alternative by attribute similarity and price delta with one-tap customer confirmation — the order is not cancelled and conversion holds. The picker app runs optimized in-warehouse routing with barcode confirmation and per-item timing, so slow zones show up as numbers instead of opinions.

The courier app is offline-first: local SQLite storage with background sync, proof of delivery by SMS code, cash handling with change calculation, on-device receipt printing. Events batch-sync without loss when the network drops.

The real constraint. The hardware was not ours to choose. Couriers use whatever phone they own, on whatever signal the street gives them, and no amount of backend quality survives that. Offline-first was forced rather than selected — every courier-side feature had to work with no network and reconcile afterwards, which is a materially harder build than the online version and the reason the delivery evidence chain holds up in a dispute.

What changed.

What was breaking. The dispatch board was wired to integrations by hand, which capped growth at the size of the ops team. Disputed deliveries had no evidence base — proof of delivery, photos and geodata were captured unreliably, so the company absorbed “not received” losses it could not disprove. Arrival times did not update when routes changed, so customers watched stale ETAs and called about them. Orders, stock and delivery events lived in three systems reconciled by hand every day.

What we built. We standardized courier integrations behind one internal interface, so a new partner became a matter of configuring credentials and an endpoint. We connected three partners directly, and the client connected a fourth themselves using the same pattern.

We strengthened the delivery evidence chain: geotagged and timestamped photo, signature, SMS code. Every delivery now carries an evidence package that closes a dispute in minutes instead of absorbing it. Dispatch and customer communications were linked, so a route change recalculates the ETA and pushes an update automatically. A data warehouse and ETL pipeline consolidated orders, stock and delivery logs into one operational view.

The real constraint. Three programs were running at once, so the recurring integration queue had to disappear rather than grow. Standardizing partner onboarding as configuration removed that queue; the client connecting the fourth partner without us proved the pattern transferred.

What changed.

What was breaking. The gap between physical reality and the digital record was closed by hand. Inspectors documented damage and checked marking across more than a thousand containers a day over three shifts, with quality depending on the individual and the lighting. Marking-versus-manifest discrepancies were caught after the fact, often after customs processing had started, and each one was a delay and a penalty. Damage disputes had no photo evidence, so the terminal regularly absorbed other parties’ claims. Off-the-shelf computer vision models were not reliable in real port conditions: fog, night, direct sun.

What we built. A software layer on the existing camera infrastructure — 127 fixed and 12 PTZ cameras at gate-in, gate-out, yards and approaches — turning video into events, evidence and workflow.

Transfer learning on YOLOv8 brought accuracy to production level. Inference runs through NVIDIA Triton, auto-saving a still frame with timestamp, container ID and camera on every detection. OCR extracts marking, ISO codes, hazmat placards and seal numbers, cross-referencing the manifest via the EDIFACT feed — a discrepancy alerts the dispatcher within 90 seconds, before customs rather than after. On exit, the system generates an annotated PDF report with confidence scores and submits it to the customs pre-clearance portal with no human involvement.

The real constraint. No usable dataset existed, and no general model was going to survive this terminal’s weather. That put three months of data collection in front of a single production inference — 45,000+ labeled images across 180+ categories of damage, marking defects and seal violations, gathered alongside the terminal’s own inspectors and tuned to this terminal’s cargo profile. It is the least impressive part of the project and the reason the model holds up in this terminal's weather.

What changed.