This page will display up to the minute service information on our TOTSCo Hub and Account Management Portal, with an explanation of the issue, impact and resolution.
We will soon be retiring this page, as the new ‘Outage Calendar' tool in the Account Management Portal now displays all planned and unplanned changes.
26/11/2024 16:00
The issue has been resolved.
26/11/2024 14:09
OTS Message Report is showing no data.
OTS Message Report is showing no data.
26/11/2024 16:00
Fix has been deployed in Prod.
03/11/2024 10.00
Fix has been deployed in Prod. Mail has been delivered to key users. Issue is under observation.
02/11/2024 00.15
We have identified the root cause of the issue and will be using the weekend to resolve and test the solution. The restoration plan will be finalized, and we will provide an update no later than Monday morning.
01/11/2024 17:15
We believe we have established the root cause of the issue and are now working on the resolution.
01/11/2024 16:08
Currently experiencing an intermittent issue where a few CPs have reported that they are not receiving outage communication.
Users are not receiving all notifications as expected and some users who should be able to are unable to access the outage calendar.
Note: Only the users with the following roles should receive email notifications of outages and be able to access the calendar: Account administrators, "Manage All" users, and technical responsibility users.
Investigation is in progress. Please check this page for updates.
RESOLVED- We're currently experiencing a data loading issue with the OTS Message Report, where some information isn't being fully retrieved or displayed. We are actively working on a fix and expect to have the issue resolved within the next 7 days.
Users may encounter difficulties when accessing or viewing their OTS Message Reports.
OTS message report issue has been resolved and report has been published for all CPs. If you have any questions please reach out to the service desk by raising a ticket.
16/9/24 8.20am
RESOLVED
Over the weekend, we identified the root cause and implemented a solution, which has been rigorously tested. As a result, the OTS Message Reporting area of the portal is now fully operational.
13/9/24 6.30pm
We have identified the root cause of the issue and will be using the weekend to resolve and test the solution. As a result, the OTS Message Reporting area of the portal will remain unavailable. To that end, we have reclassified this issue as a P2. The restoration plan will be finalised, and we will provide an update no later than Monday morning.
13/9/24 4.30pm
We are aware of a technical issue within the Account Management Portal, while we work to understand and resolve the issue, you will be unable to access the OTS message reporting area of the Account Management Portal.
RESOLVED
RESOLVED
RESOLVED: Switch match requests in pre-production are not flowing through
Users of the pre-production environment faced an issue resulting in a failure status on switch match requests from 11:19 to 14:10 hours BST. CPs were required to resubmit the messages.
The issue has been identified and fixed.
RESOLVED - We are aware of a technical issue within the ticketing functionality of our Account Management Portal. While we work on resolving the issue, the ticketing system is temporarily unavailable.
Ticketing system is temporarily down, please log a ticket vis the service desk - [email protected]
RESOLVED - The issue has been identified and fixed
Issue fixed - Following further investigation, we have reclassified the issue as Priority 2, as the majority of messages are flowing through. Where we have identified issues with individual messages, we are reaching out to the organisations impacted to reach a resolution.
Users in integration Testing
We are closing the P2 issue as the hub has been restored. We have now opened a problem ticket to continue our investigations into the incident.
RESOLVED - Target CP will not receive messages from source CP
Users in Integration Testing
RESOLVED - The issue was identified and fixed.
RESOLVED - Issue with the DNS impacting all environments
Users testing
RESOLVED - The issue was found, investigated and fixed in quick succession.
RESOLVED - Issue of occasional messages not being delivered.
Some users in integration testing. Users in this environment can continue to test.
After investigating, it appears that the issue lies on the CP side. We are currently collaborating with all CPs to resolve these issues effectively. Update 28/03/2024
RESOLVED - Intermittent fault has been identified where duplicate messages are being sent to users. This is currently under investigation.Some users in integration testing. Users in this environment can continue to test.
Some users in integration testing. Users in this environment can continue to test.
RESOLVED - Fix was successfully tested in system integration testing (SIT) and deployed successfully. Update 20/03/24
RESOLVED - Message response issue where multiple messages are sent by one user to another (one RCPID to another RCPID). We are aware of an issue when sending multiple messages between the same two users in the Integration Test Environment. In this situation, the responses to the receiving user were for a sub-set of the messages from the Hub, and responses were pending for the remaining messages.
Users sending multiple messages between each other in the Integration Test Environment.
THIS ISSUE HAS BEEN RESOLVED. We have fine-tuned one of the configuration parameters in the Hub to fix the issue.
RESOLVED: We currently have an issue in our TOTSCo Hub integration testing environment. The hub is not facilitating message exchanges.
All users currently testing in the integration environment.
Now resolved. Issue was due to an access key failure. Restoration was achieved by creating a new key. The issue was moved to the problem stage (as per ITIL processes) for further root cause analysis.
RESOLVED: The account management portal went down.
All CPs using the account management portal.
Now resolved. If you are unable to access your account please let us know by contacting [email protected]
RESOLVED: Integration testing environment unavailable.
All CPs using the integration testing environment.
A full investigation was carried out the issue was related to our Domain Name System (DNS) settings. Upon investigation we learnt that the issue predominantly affected our IT partner, Tech Mahindra. The integration testing environment is now available. You can now use the Integration Test Environment as normal.
CPs receiving 503 response intermittently.
CPs getting failure messages on intermittent basis resulting in not able to exchange message.
18/12/2023: Service Desk informed affected CPs that the issue has now been resolved. 15/12/2023: Issue resolved on Friday at 8:30pm GMT. The TOTSCo technical team investigated the intermittent (Error 503) on hub. There was a configuration issue with one of the load balancer nodes which was identified and corrected to resolve this issue. 11/12/2023: under investigation
Test simulator down.
3 hour outage of the Test Simulator.
1pm 20/11/2023. Additional memory allocated to the Test simulator. Issue resolved.
