Resolved Email delivery delays via SendGrid west region
May 14, 2026A regional SendGrid outage caused a 4-12 minute delay on outbound email campaigns originating from us-west-2. AWS SES failover engaged automatically at 14:08 UTC, delivery returned to normal by 14:31 UTC. No emails were lost - all queued sends were delivered.
14:31 UTCResolvedAll queued sends delivered. Returning to SendGrid west region.
14:18 UTCMonitoringSES failover stable. Watching for backlog drain.
14:08 UTCInvestigatingSendGrid west region returning 502s. Engaging AWS SES failover.
Resolved Dutchie webhook delivery delays
May 8, 2026Dutchie's webhook origin saw elevated latency between 21:14-21:47 UTC. Our ingestion queue absorbed the spike and replayed all events on recovery. Loyalty points were applied correctly with a 30-minute lag for ~7,800 orders.
21:47 UTCResolvedDutchie webhook flow back to normal. All delayed events processed.
21:22 UTCMonitoringQueue backlog draining at ~280 events/sec. No customer-facing errors.
21:14 UTCInvestigatingElevated latency on Dutchie webhook origin. Customer ingestion buffered.
Maintenance Database read-replica region addition (eu-central-1)
Apr 27, 2026Added a Postgres read replica in eu-central-1 to reduce dashboard load times for European customers. Two-hour scheduled window completed in 1h 24m with no customer impact.
04:24 UTCCompletedeu-central-1 read replica live. Dashboard P95 in EU now <180ms.
03:00 UTCStartedRead replica provisioning begins. No write or read path changes.
Resolved Wallet pass refresh queue lag
Apr 19, 2026After a large promotional send by a multi-location customer, wallet pass refresh saw a 6-minute queue lag. Auto-scaling kicked in, queue drained to normal by 18:33 UTC. Customer passes refreshed within 12 minutes of the original send (vs. the typical <30s).
18:33 UTCResolvedPass refresh queue back to <30s. New auto-scale floor raised.
18:14 UTCInvestigatingSingle customer's 18k-pass refresh caused queue lag. Auto-scaling triggered.
Maintenance Sticky AI Playbooks v3 launch
Apr 12, 2026Shipped Sticky AI Playbooks v3, doubling the number of pre-built journey templates from 6 to 12. Existing customer playbooks were not affected; new templates surfaced in-app immediately.
Resolved SMS carrier filtering on T-Mobile traffic
Apr 3, 2026Brief T-Mobile-side filter rule change caused a ~14% block rate on outbound cannabis SMS for 38 minutes. Worked with T-Mobile's MNO operations team to lift the filter. Affected sends were re-queued automatically. Customer dashboards show flagged sends with a recoverable status.
19:42 UTCResolvedT-Mobile filter rule reverted. Block rate back to baseline.
19:04 UTCInvestigating14% block rate detected on T-Mobile destinations. Escalating with MNO ops.
