Get email notifications whenever EdSmart Platform Status creates, updates or resolves an incident.
Get text message notifications whenever EdSmart Platform Status creates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Visit our support site.
All Systems Operational
API
Operational
Application
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Website Uptime
Fetching
API Uptime
Fetching
Website Response Time
Fetching
API Response Time
Fetching
Past Incidents
Aug 18, 2022
No incidents reported today.
Aug 17, 2022
No incidents reported.
Aug 16, 2022
Postmortem -
Read details
Aug 17, 19:06 AEST
Aug 17, 19:06 AEST
Resolved -
This incident has been resolved.
Aug 16, 16:35 AEST
Aug 16, 16:35 AEST
Update -
(Editing to change the Incident Status to Resolved.)
Service is now restored on the EdSmart Platform. We are awaiting a Root Cause Analysis (RCA) from the Microsoft Azure Support Team to explain how their infrastructure caused this significant outage whilst preventing us from activating failover processes.
We will continue to monitor and work to implement measures to eliminate or reduce the impact of this problem should it happen again.
Please continue to subscribe to https://status.edsmart.com for updates.
If you are still experiencing issues, or would like a detailed RCA, please contact help@edsmart.com.
We apologise for the inconvenience this may have caused and appreciate your patience and understanding today.
Aug 16, 16:34 AEST
Service is now restored on the EdSmart Platform. We are awaiting a Root Cause Analysis (RCA) from the Microsoft Azure Support Team to explain how their infrastructure caused this significant outage whilst preventing us from activating failover processes.
We will continue to monitor and work to implement measures to eliminate or reduce the impact of this problem should it happen again.
Please continue to subscribe to https://status.edsmart.com for updates.
If you are still experiencing issues, or would like a detailed RCA, please contact help@edsmart.com.
We apologise for the inconvenience this may have caused and appreciate your patience and understanding today.
Aug 16, 16:34 AEST
Update -
Service is now restored on the EdSmart Platform. We are awaiting a Root Cause Analysis (RCA) from the Microsoft Azure Support Team to explain how their infrastructure caused this significant outage whilst preventing us from activating failover processes.
We will continue to monitor and work to implement measures to eliminate or reduce the impact of this problem should it happen again.
Please continue to subscribe to https://status.edsmart.com for updates.
If you are still experiencing issues, or would like a detailed RCA, please contact help@edsmart.com.
We apologise for the inconvenience this may have caused and appreciate your patience and understanding today.
Aug 16, 16:24 AEST
We will continue to monitor and work to implement measures to eliminate or reduce the impact of this problem should it happen again.
Please continue to subscribe to https://status.edsmart.com for updates.
If you are still experiencing issues, or would like a detailed RCA, please contact help@edsmart.com.
We apologise for the inconvenience this may have caused and appreciate your patience and understanding today.
Aug 16, 16:24 AEST
Update -
Updating to reflect that API and Application services are now running.
Aug 16, 14:30 AEST
Aug 16, 14:30 AEST
Monitoring -
Our database is now running and we are restoring services on the EdSmart Platform and running diagnostic tests. You may experience some performance issues while systems start back up.
Aug 16, 14:28 AEST
Aug 16, 14:28 AEST
Update -
Updating outage information to show that the API is unavailable in addition to the application.
Aug 16, 14:15 AEST
Aug 16, 14:15 AEST
Update -
Microsoft Azure Support Engineers are working to cancel the stalled auto scaling operation. We are awaiting further updates from the Azure Support team and will continue to provide updates via https://status.edsmart.com.
Aug 16, 14:10 AEST
Aug 16, 14:10 AEST
Identified -
The issue lies with the scale up/down functions in the Azure database platform. This is functionality we have been using without incident for some 8 years. The most recent automated scale up operation stalled and our engineering team were not able to manually resolve.
We have found some references online, indicating that this is a very rare but known issue with Azure SQL.
The nature of the issue also appears to have sidestepped our monitoring, for example, database connection monitoring, which did not trigger. It seems Azure 'thinks' the database is available when actually it isn't.
The engineering team is dealing directly with senior account and engineering management at Microsoft Azure. In the meantime, they are also working to make the database available via other means, as the accepted Azure restore methods for failover are also unavailable due to the current issue.
Aug 16, 12:55 AEST
We have found some references online, indicating that this is a very rare but known issue with Azure SQL.
The nature of the issue also appears to have sidestepped our monitoring, for example, database connection monitoring, which did not trigger. It seems Azure 'thinks' the database is available when actually it isn't.
The engineering team is dealing directly with senior account and engineering management at Microsoft Azure. In the meantime, they are also working to make the database available via other means, as the accepted Azure restore methods for failover are also unavailable due to the current issue.
Aug 16, 12:55 AEST
Update -
Microsoft Azure Support are still investigating the issue, it’s ranked at the highest priority in their support system and we are in constant contact with their account and technical management teams.
Aug 16, 11:44 AEST
Aug 16, 11:44 AEST
Update -
Microsoft Azure Support have escalated management for this critical issue and are working with EdSmart to identify and resolve the problem urgently. Please continue to subscribe to https://status.edsmart.com/ for updates. Thank you for your understanding.
Aug 16, 10:06 AEST
Aug 16, 10:06 AEST
Investigating -
Our Support and Engineering teams are aware of an issue with our Microsoft Azure SQL Database as of 6:30 am local time. We are investigating the issue with highest priority and have contacted Microsoft Support.
Please continue to subscribe to status.edsmart.com for updates. We apologise for the inconvenience this may have caused and we endeavour to restore service as soon as possible. We appreciate your patience and understanding during this time.
Aug 16, 08:51 AEST
Please continue to subscribe to status.edsmart.com for updates. We apologise for the inconvenience this may have caused and we endeavour to restore service as soon as possible. We appreciate your patience and understanding during this time.
Aug 16, 08:51 AEST
Aug 15, 2022
No incidents reported.
Aug 14, 2022
No incidents reported.
Aug 13, 2022
No incidents reported.
Aug 12, 2022
No incidents reported.
Aug 11, 2022
No incidents reported.
Aug 10, 2022
No incidents reported.
Aug 9, 2022
No incidents reported.
Aug 8, 2022
No incidents reported.
Aug 7, 2022
No incidents reported.
Aug 6, 2022
No incidents reported.
Aug 5, 2022
No incidents reported.
Aug 4, 2022
No incidents reported.