Shipup - Notice history

We wanted to take the time to provide a transparent summary of an incident that affected our system from January 6th to the 14th. During this period, we experienced difficulties in retrieving package events and sending associated notifications. The root cause of the issue was an ongoing upgrade to our database, which inadvertently put excessive read/write operations on the disk. This slowdown significantly impacted critical operational processes, including event retrieval and notification scheduling. Before starting the migration, we deliberately targeted a low-activity period to minimize disruptions to our customers' experience. The production migration started on December 29th, but at first, we didn't see any issues with the system. However, it wasn't until January 6th that the processing of large tables caused all database operations to slow down dramatically. Our team worked diligently to optimize operations and restore normal system performance over the course of several days. We initially believed that notifications were being sent despite delays in tracker event processing, only to discover on Thursday 9th later that no notifications had been sent since Tuesday 7th morning. We quickly addressed this issue and restored notification functionality on Thursday evening. The migration operation was completed on January 14th, and our system is now back to normal regarding event retrieval and notification sending. Lessons learned: * We recognized that internal alerts didn't provide accurate information twice during the incident. We're working to resolve these issues and ensure no false-positive alarms in the future. * Although testing concluded before the migration, we acknowledge the importance of warning customers about ongoing changes, even if it's just a heads-up or a scheduled maintenance window. This would have allowed for more proactive communication and potentially minimized the impact. Our goal is to provide our customers with a reliable and trustworthy experience. We take incidents like this seriously and are continually working to improve our processes and communication channels.

The incident has ended on the 14/01 around 4pm. After monitoring the situation, we confirm that this incident is now closed. A Post Mortem will be shared here soon.

Huge improvement has been noticed. We are monitoring the situation closely and we will send an email today to share news about this issue.

Huge improvement has been noticed. We are monitoring the situation closely and we will send an email today to share news about this issue.

We are still experiencing difficulties in retrieving carrier events. The retrieval frequency is less regular and more spaced out. In terms of impact : - Shipup notifications are sent with a delay. They may not be sent at all if they fall within our obsolescence rules. - Shipup data analysis is momentarily impacted. - Once the problem is solved, we will recover all events. The data willl therefore be available for analysis shortly. We expect to be back to normal state tomorrow evening.

We are continuing to work on a fix for this issue. The return to normal should occur by Tuesday, January 14th.

We are continuing to work on a fix for this issue.

- Shipup notifications are sent with a delay. They may not be sent at all if they fall within our obsolescence rules. - No notifications were sent between January 7 and January 9. Notifications have been sent again since today, including, progressively, for orders placed in the last few days. - Shipup data analysis is momentarily impacted. - Once the problem is solved, we will recover all events. The data will therefore be available for analysis shortly. IMPORTANT: your clients are still receiving the notifications sent by the carriers (even if you have the "Filter carrier notifications" on - we have disabled it until the resolution). Therefore, your clients receive the tracking status of their orders.

We are experiencing difficulties in retrieving carrier events. The retrieval frequency is less regular and more spaced out. In terms of impact : - Shipup notifications are sent with a delay. They may not be sent at all if they fall within our obsolescence rules. - Shipup data analysis is momentarily impacted. - Once the problem is solved, we will recover all events. The data willl therefore be available for analysis shortly. We expect to be back to normal state tomorrow evening.

We are recovering all ongoing shipment tracking events again. Some customers may have received follow-up notifications with a few hours of delay. We apologize for the inconvenience caused.

The problem has been identified and the system is gradually returning to a normal state since this afternoon. We will keep you informed when a complete return to normalcy is confirmed.