Mar 17, 17:01:00 GMT+0
Resolved -
This incident has been resolved..
Mar 17, 16:35:00 GMT+0
Investigating -
We are currently investigating this incident..
Mar 6, 18:28:34 GMT+0
Resolved -
This incident has been resolved..
Mar 6, 17:56:20 GMT+0
Identified -
We've identified the issue and have verified most services are back to full functionality. We're still working on resolving issues with the METAR service..
Mar 6, 17:39:00 GMT+0
Investigating -
We are currently investigating this incident..
Mar 6, 18:01:32 GMT+0
Monitoring -
All services are back to full functionality. We'll continue monitoring for a bit to ensure stability..
Mar 2, 23:41:46 GMT+0
Investigating -
We are currently investigating this incident..
Mar 2, 23:53:17 GMT+0
Monitoring -
We implemented a fix and are currently monitoring the result..
Mar 3, 00:15:00 GMT+0
Resolved -
This incident has been resolved..
Feb 22, 17:24:00 GMT+0
Monitoring -
Due to a denial of service attack against our infrastructure, some services may be degraded until the attack ceases. If you encounter errors logging into the network or myVATSIM, please be patient and try again..
Feb 23, 01:40:22 GMT+0
Resolved -
The attack has lifted as of a couple hours ago. We'll continue monitoring for further impact, but at this point the incident is resolved..
Jan 18, 19:29:00 GMT+0
Identified -
Our upstream data provider for METARs is returning unexpected data. We're working on implementing a workaround..
Jan 18, 22:59:57 GMT+0
Monitoring -
We've now implemented a fix to get METARs back in order. We'll monitor it for the next while to confirm it's working properly..
Jan 18, 23:55:34 GMT+0
Resolved -
This incident has been resolved..
Jan 10, 21:55:00 GMT+0
Resolved -
This incident has been resolved..
Jan 9, 00:00:00 GMT+0
Completed -
Sweatbox-1 will be rebooted tonight at 0000Z
.
Jan 4, 03:00:14 GMT+0
Resolved -
This incident has been resolved..
Jan 4, 02:43:44 GMT+0
Investigating -
We are currently investigating this incident..
Jan 4, 02:50:11 GMT+0
Monitoring -
We implemented a fix and are currently monitoring the result..
Nov 10, 13:29:38 GMT+0
Identified -
We are aware of an upstream issue with our METAR provider (US NOAA). We will post an update when the issue is resolved..
Nov 10, 19:14:30 GMT+0
Resolved -
This incident has been resolved..
Oct 30, 05:52:27 GMT+0
Completed -
Maintenance has concluded..
Oct 30, 05:00:00 GMT+0
Identified -
Kubernetes upgrades will occur tonight at 05:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade..
Oct 30, 05:00:41 GMT+0
Identified -
Maintenance is now in progress..
Oct 24, 05:24:00 GMT+0
Identified -
USA-WEST FSD is unable to communicate with its hub due to networking issues at our compute provider. Various systems are impacted..
Oct 24, 06:13:16 GMT+0
Identified -
Our compute provider has acknowledged the issue, for updates on their end please see here: https://status.digitalocean.com/incidents/gp9bzm1hnlk4.
Oct 24, 06:16:09 GMT+0
Identified -
We have moved the Dataserver to another region and cordoned USA-WEST. The network is operating at reduced capacity until we have confirmation that the upstream issue is resolved..
Oct 24, 14:17:47 GMT+0
Resolved -
Our upstream provider has resolved the issue.
We have enabled traffic to USA West..
Oct 16, 20:30:06 GMT+0
Monitoring -
We've now implemented a fix and will be monitoring for a while longer..
Oct 16, 21:52:40 GMT+0
Resolved -
This incident has been resolved..
Oct 16, 19:00:00 GMT+0
Investigating -
Requests to forum.vatsim.net are returning the error "no healthy upstream" for the majority of our users. We are currently investigating this incident..
Oct 16, 16:33:55 GMT+0
Identified -
USA METARs are currently out of date due to changes at AviationWeather.gov. We’re working on a fix and will update this incident once resolved..
Oct 16, 17:10:14 GMT+0
Resolved -
We've updated our services to use the new AWC APIs and are now seeing recent USA METARs populate across the network. This incident is now resolved..
Oct 14, 11:30:08 GMT+0
Resolved -
This incident has been resolved..
Oct 14, 09:52:48 GMT+0
Investigating -
We are currently investigating potential issues with VATSIM network connectivity. Please stay tuned for updates..
Oct 14, 10:39:57 GMT+0
Investigating -
We are continuing to investigate this incident. We are also aware one of our upstream providers is having a network issue, which may be contributing to our connectivity problems. Updates will follow..
Oct 14, 11:20:44 GMT+0
Monitoring -
We are seeing recovery of all connections at this time. We believe this was related to an incident at an [upstream provider.](https://status.digitalocean.com/incidents/bhpslzd37517). .
Sep 19, 22:51:49 GMT+0
Resolved -
This incident has been resolved..
Sep 19, 16:50:35 GMT+0
Identified -
METARs for airports in the United States are currently out-of-date due to issues with the Aviation Weather Center. We're monitoring the upstream issue and will close this incident once we have observed recovery..
Sep 13, 20:54:56 GMT+0
Identified -
One of our upstream providers is experiencing networking problems in their SFO region. This is currently affecting FSD's USA-WEST server..
Sep 13, 21:00:52 GMT+0
Identified -
FSD USA-WEST has been taken out of service while connectivity is restored..
Sep 13, 22:00:51 GMT+0
Identified -
FSD USA-WEST and a server supporting Automatic Server Selection have been disconnected from the network. This removes impact from network operations. Once connectivity is restored we will bring them back into service. At this time everything else is operating properly. New connections will be sent to servers not impacted and existing have been migrated to others.
When those servers were in service, users may have occasionally been unable to connect to FSD or experienced netsplits..
Sep 13, 23:48:58 GMT+0
Identified -
We are continuing to keep FSD USA-WEST and one server supporting Automatic Server Selection out of service. All other systems remain fully operational..
Sep 14, 01:53:58 GMT+0
Resolved -
This incident has been resolved. FSD USA-WEST and Automatic Server Selection are fully operational..
Sep 14, 01:26:06 GMT+0
Monitoring -
We plan to bring US-WEST and Automatic Server Selection back to full service shortly. Network connectivity issues with one of our providers has been resolved..
Aug 29, 02:12:44 GMT+0
Monitoring -
We implemented a fix and are currently monitoring the result..
Aug 29, 02:46:39 GMT+0
Resolved -
This incident has been resolved..
Aug 29, 01:03:11 GMT+0
Identified -
A temporary fix has been put in place. We continue to implement a permanent solution..
Aug 29, 00:30:50 GMT+0
Identified -
Currently https://metar.vatsim.net is returning 500 errors. We have identified the problem and are working to resolve it..
Aug 22, 16:24:35 GMT+0
Monitoring -
We're starting to see recovery at this time. The Datafeed is back to its normal state and AFV connections are no longer being timed out.
We'll keep monitoring this incident in case any further issues come up..
Aug 22, 14:00:00 GMT+0
Identified -
Our compute provider is experiencing networking issues that are affecting our Datafeed and Slurper systems.
VAT-Spy, Simaware, and other consumers of the Datafeed may occasionally receive incorrect or corrupt data.
Pilots and controllers may occasionally become disconnected from Audio for VATSIM.
Thank you for your patience while we work towards a resolution with our vendors..
Aug 22, 16:41:23 GMT+0
Identified -
While the issues with Slurper have resolved, we're still seeing occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data..
Aug 22, 17:26:55 GMT+0
Identified -
We are currently seeing network connectivity issues at our provider's datacenter in Frankfurt. Pilots and controllers connected to Germany 1 & 2 may occasionally be timed out by the AFV Slurper and lose audio connection.
We've also observed occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data..
Aug 22, 20:47:57 GMT+0
Monitoring -
We've restarted more of our systems and are seeing recovery. Please reach out if you're still seeing issues..
Aug 22, 21:37:31 GMT+0
Resolved -
This incident has been resolved..
Aug 9, 15:03:30 GMT+0
Resolved -
This incident has been resolved. Our compute provider has patched their networking systems across all regions and has observed full recovery..
Aug 9, 09:55:00 GMT+0
Identified -
Our compute provider for FSD and other services is currently experiencing issues globally across their network.
Users may occasionally be unable to connect to FSD or experience netsplits.
We will update this incident once we have confirmation that the networking issues have been resolved..
Aug 9, 13:34:46 GMT+0
Monitoring -
Our compute provider has deployed a fix to all regions. Recovery may still be ongoing as systems return back to normal.
We'll continue monitoring this issue in case it crops back up..
Jul 9, 13:09:00 GMT+0
Identified -
One of our Kubernetes nodes went offline unexpectedly and has caused several web services to go down. We're working on bringing everything back up..
Jul 9, 13:25:03 GMT+0
Resolved -
Access to web services has been restored - this incident is now resolved..
Jul 4, 03:00:00 GMT+0
Identified -
We will be performing a migration of support.vatsim.net to a new server at 0300z.
The Support Center will be unavailable during that time..
Jul 4, 03:01:21 GMT+0
Identified -
Maintenance is now in progress. The Support Center will temporarily be unavailable while we complete a migration..
Jul 4, 03:28:57 GMT+0
Completed -
The support center migration has completed.
If you notice any issues, please reach out to the Technology team via [support.vatsim.net](https://support.vatsim.net/open.php)..
Jun 22, 03:00:00 GMT+0
Identified -
We will be performing a migration of learn.vatsim.net to a new server at 0300z.
The E-Learning system will be unavailable during that time..
Jun 22, 03:50:32 GMT+0
Identified -
Maintenance is now in progress.
Due to the late start, it may last longer than originally estimated..
Jun 22, 04:27:09 GMT+0
Completed -
The great E-Learning migration has completed.
If you notice any issues, please reach out to the Technology team via [support.vatsim.net](https://support.vatsim.net/open.php)..
May 16, 04:00:00 GMT+0
Identified -
Maintenance is now in progress..
May 16, 04:13:00 GMT+0
Completed -
Maintenance has completed successfully..
May 16, 04:00:00 GMT+0
Identified -
Kubernetes upgrades will occur tonight at 04:00 UTC. Expect web services to be flaky while our compute instances are rotated out..
May 14, 20:09:00 GMT+0
Monitoring -
We've restarted our API deployment and are starting to see recovery at this time..
May 14, 15:47:00 GMT+0
Investigating -
We are currently investigating this incident..
May 14, 22:27:00 GMT+0
Resolved -
This incident has been resolved..
May 10, 22:41:00 GMT+0
Resolved -
This incident has been resolved..
May 10, 19:36:00 GMT+0
Investigating -
We are currently investigating this incident..
May 10, 19:49:00 GMT+0
Monitoring -
We have deployed a mitigation to myVATSIM that should resolve the issue. We will continue to monitor the issue and provide updates as they become available..
Apr 1, 12:36:00 GMT+0
Investigating -
We are investigating database connectivity issues. This is causing some users to be unable to connect to the network and our web services. We will update this incident as progress is made..
Apr 1, 12:46:00 GMT+0
Monitoring -
We have made some back end changes to assist and things appear to be recovering. We are still monitoring..
Apr 1, 13:39:00 GMT+0
Resolved -
This incident has been resolved..
Mar 8, 02:00:00 GMT+0
Identified -
We plan to apply updates to our database cluster during this maintenance window. No user impact is expected during this period of time. We will keep this page updated during the maintenance window..
Mar 8, 02:59:00 GMT+0
Identified -
We have started database maintenance. We will keep this page updated if any issues arise..
Mar 8, 03:57:00 GMT+0
Completed -
Maintenance has completed successfully..
Jan 31, 01:41:00 GMT+0
Identified -
Maintenance is now in progress..
Jan 31, 02:27:00 GMT+0
Completed -
Maintenance has completed successfully..
Jan 21, 01:49:00 GMT+0
Investigating -
We are investigating network connectivity issues on multiple FSD servers..
Jan 21, 02:08:00 GMT+0
Monitoring -
Connectivity has returned to normal and all FSD servers are functioning. We are continuing to monitor the network..
Jan 21, 02:21:00 GMT+0
Resolved -
This incident has been resolved..
Jan 21, 01:59:00 GMT+0
Identified -
We have shifted some connections around to resolve the impact on several servers..