Shipup - Notice history

This incident has been resolved. We're back at receiving 100% of the USPS events.

We are able to subscribe to USPS webhooks again. We'll keep monitoring and post another update once we are fully recovered

A fix has been implemented, and Canada Post trackers are now correctly tracked again since September 10.
At the time of posting this update (September 18), we've confirmed that the situation seems stable.

This incident has been fully resolved.

Description

Date: May 15th 2026

We experienced an issue affecting carrier event processing for Chronopost and Colissimo shipments.

The incident started on May 4th 2026, but was only detected later after reports of delayed tracking updates, as it increased over time. The issue originated from our Event Fetching V3 system, where carrier events were being processed with increasing delay.

On May 13th, we mitigated the issue by temporarily rerouting processing from Event Fetching V3 to our legacy Event Fetching V2 system, which reduced the lag and restored normal processing for Chronopost shipments.

On May 15th, a similar issue reoccurred and affected Colissimo shipments as well. The same mitigation was applied to reduce the impact.

The mitigations implemented during the incident were temporary measures. We are continuing to investigate the root cause and work on a permanent solution.

Business consequences of the issue

During the incident period, some merchants experienced delayed shipment tracking updates for Chronopost and Colissimo deliveries.

As a consequence:

• Shipment notifications could be delivered later than expected.

• In some cases, notifications may not have been sent if carrier events exceeded internal expiration thresholds.

• Shipment tracking visibility may have been temporarily degraded.

The issue primarily affected shipments using Chronopost and Colissimo integrations.

The absence of proper monitoring on event-processing lag delayed the detection of the issue and increased the duration of customer impact.

Chronology

• May 4th: Increased lag begins in carrier event processing on Event Fetching V3.

• May 12th: First reports of delayed tracking updates are received.

• May 13th: Investigation confirms lag affecting Chronopost event processing.

• May 13th: Mitigation applied by rerouting processing from Event Fetching V3 to Event Fetching V2.

• May 15th: Similar lag issue detected affecting Colissimo event processing.

• May 15th: Same mitigation strategy applied to reduce impact.

Learnings

• We lacked proper monitoring and alerting on carrier event-processing lag.

• The issue remained undetected for several days before customer reports triggered the investigation.

• Existing fallback procedures using the legacy V2 system helped reduce impact, but are not a sustainable long-term solution.

Short term actions

Description of the short-term actions that could help prevent recurrence:

• Add metrics and alerting on carrier event-processing lag

• Improve operational visibility on Event Fetching V3 processing queues

• Document and standardize mitigation procedures for rerouting processing traffic

Long term actions

Description of the long-term actions:

• Investigate and resolve the root cause of lag generation in Event Fetching V3

• Improve scalability and resilience of the event-processing pipeline

• Review notification expiration rules and evaluate fallback mechanisms for delayed events

• Reduce dependency on the legacy V2 system for incident mitigation

We took a mitigation action that will stabilise the tracking update latency. Trackers are getting up to date as we monitor.

Sendgrid emails do now go through. Emails that were held up during the outage will be sent automatically by Sendgrid.

Carrier notifications are still being exceptionally sent.

We are continuing to work on a fix for this incident.

The reason that led SendGrid to lock our account has been solved. We are now waiting for SendGrid to unlock our account to be back on normal.

To be noted: none of this incident has been caused by a security issue, but only by a false classification by a Spam listing organization.

We identified the reason SendGrid blocked our account, and this reason is being corrected.

We also requested the SendGrid Compliance Team to unlock our account; we are waiting for their answer.

Please note: all the Carrier Notifications are sent correctly and have not been disturbed.

During the incident, all the filtered Carrier Notifications are exceptionally sent, to make sure none of the Shoppers miss an important update about their packages.

The root cause of the incident was identified and a complete fix deployed.

We took mitigation measures and are monitoring as performances go back to normal.

We are working closely with the carrier to implement a new integration method.

We implemented a fix and are currently monitoring the result.

We started updating pickup point information again on 07/05 4pm UTC