Turvo Status
Affected services:
Opened on Tuesday 5th October 2021, last updated
Resolved — As of approximately 3pm EST on 06OCT21 we are confirming that the original incident with the AWS-hosted RabbitMQ message queueing system has been fully resolved and all systems are fully operational. The Turvo teams continue to ensure the backlog of shipments and orders, and other data elements for various customers are processed from end-to-end. Thank you very much for your patience.
Monitoring — All message queue processing supporting shipments, orders and other integration data remains stable, however we continue to monitor the situation closely with the highest priority. We are working directly with select integration customers to validate that all of their backlog messages are reflowed and processed to ensure end-to-end data consistency from the beginning of the incident which started approximately 6:30pm EST on 05OCT21.
Monitoring — All our backlog on queues has been processed. We are working with select integration customers to validate that all of their backlog messages are processed.
Monitoring — We have restored our integration services, however a backlog of queued messages needs to be processed and select integration customers will be contacted to reflow a subset of their messages if needed
Investigating — We are upgrading the severity of this issue at this time due to ongoing Integration pipeline issues which has impacted the incoming flow of shipments, orders and other integration data elements.
Monitoring — There is improvement in performance we are seeing as the environment has started stabilizing. We will continue monitor. Stay tune for further updates !
Identified — While mitigating this issue we are stabilizing our infrastructure and we have made some more progress. We are constantly monitoring and working on the issue.