Duration: 19 hours and 33 minutes

Affected Components: , , , , , , , , , , , , , , ,

Sep 30, 11:58:18 GMT+0
Resolved - This incident has been resolved..

Sep 29, 18:46:07 GMT+0
Monitoring - The attack has reduced for the past few hours. However, it may resume at any moment depending on the whims of the attacker. We will continue to monitor..

Sep 29, 16:25:13 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 days, 11 hours and 55 minutes

Affected Components: , , , , , , , , , ,

Aug 5, 03:05:41 GMT+0
Resolved - This incident has been resolved..

Aug 3, 06:00:00 GMT+0
Monitoring - Around 6z this morning, another DDoS attack began targeting our infrastructure, resulting in servers being blackholed again. Degraded performance or loss of connectivity should be expected whilst the attack is ongoing. We're continuing to investigate mitigation methods..

Aug 3, 00:03:00 GMT+0
Monitoring - The attack has reduced for the past few hours. However, it may resume at any moment depending on the whims of the attacker. We will continue to monitor..

Aug 2, 15:11:01 GMT+0
Identified - Several of our services have been experiencing a wide-scale DDoS attack. This means many many computers around the world are sending network traffic to VATSIM in order to overwhelm our systems. Due to the size of this attack (we've seen it hit upwards of 30 Mpps, for those who know what that means), our infrastructure providers have to temporarily "blackhole" all traffic in/out of VATSIM, causing multiple servers to appear "down." We're looking into ways to further mitigate this..

Aug 2, 15:11:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 53 minutes

Affected Components: , , , , , , ,

Jul 15, 19:31:28 GMT+0
Resolved - This incident has been resolved..

Jul 15, 19:10:09 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 15, 18:38:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours and 43 minutes

Affected Components: , , , ,

Jun 23, 16:42:58 GMT+0
Completed - Upgrade is complete.

Jun 23, 15:25:57 GMT+0
Identified - Most services are restored, Bot/ATIS are still pending a fix..

Jun 23, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 23, 14:00:00 GMT+0
Identified - Audio for VATSIM will be down for an hour for a server upgrade..

Duration: 12 hours and 37 minutes

Affected Components: ,

Feb 9, 06:22:37 GMT+0
Resolved - This incident has been resolved..

Feb 8, 18:01:35 GMT+0
Monitoring - We've reset various AFV components and are monitoring. Users may have to re-connect their clients to restore audio..

Feb 8, 17:46:00 GMT+0
Identified - .

Duration: 9 hours and 57 minutes

Affected Components:

Oct 30, 01:41:48 GMT+0
Resolved - This incident has been resolved..

Oct 29, 15:45:00 GMT+0
Identified - Due to an issue with our upstream weather provider, METARs may currently be out of date or unavailable..

Duration: 24 minutes

Affected Components: ,

Sep 11, 17:45:00 GMT+0
Resolved - This incident has been resolved..

Sep 11, 17:34:54 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. We apologize for the repeated issues with Audio recently..

Sep 11, 17:26:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Sep 11, 17:21:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 22 minutes

Affected Components: ,

Sep 10, 17:09:00 GMT+0
Resolved - This incident has been resolved..

Sep 10, 16:47:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 3 days and 11 minutes

Affected Components: ,

Sep 7, 16:53:00 GMT+0
Resolved - This incident has been resolved..

Sep 7, 16:42:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 6 minutes

Affected Components: ,

Sep 6, 18:13:00 GMT+0
Resolved - This incident has been resolved..

Sep 6, 18:07:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 29 minutes

Affected Components: , , , , , , , , , , , ,

Jul 22, 04:39:33 GMT+0
Completed - Maintenance has completed successfully..

Jul 22, 04:10:35 GMT+0
Identified - Maintenance is now in progress..

Jul 22, 04:00:00 GMT+0
Identified - Kubernetes upgrades will occur July 22nd at 04:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade..

Duration: 4 days, 2 hours and 27 minutes

Affected Components: , , , , , , , , , , ,

Jul 5, 18:57:23 GMT+0
Resolved - This incident has been resolved..

Jul 1, 16:30:30 GMT+0
Investigating - Due to upstream provider network issues, the USA-E2 FSD server is currently unavailable. All other servers are unaffected..

Duration: 1 hour and 19 minutes

Affected Components:

Jun 25, 23:08:42 GMT+0
Resolved - This incident has been resolved..

Jun 25, 21:50:00 GMT+0
Identified - Due to an issue with an upstream piece of software, a regular upgrade of the Learning Center has failed. We're working on getting things back up and running..

Duration: 4 minutes

Affected Components:

May 27, 03:04:14 GMT+0
Completed - Maintenance has completed successfully..

May 27, 03:00:01 GMT+0
Identified - Maintenance is now in progress.

May 27, 03:00:00 GMT+0
Identified - The AFV Bot server will be rebooted for an upgrade. Total downtime will be approximately 10 minutes..

Duration: 4 hours and 13 minutes

Affected Components: , , , , , , , , , , , , ,

Apr 26, 15:43:36 GMT+0
Resolved - This incident has been resolved..

Apr 26, 11:32:37 GMT+0
Monitoring - We have made some infrastructure modifications to handle the high load and service appears to be recovering. We will continue to monitor as the day progresses. .

Apr 26, 11:30:29 GMT+0
Investigating - The VATSIM Tech Team is working to resolve connectivity issues to the network and some related systems caused due to high loads. Updates will continue to be posted..

Duration: 24 minutes

Affected Components: ,

Mar 14, 16:19:42 GMT+0
Resolved - This incident has been resolved. An unexpected database outage caused the Audio for VATSIM API to stall and timeout clients..

Mar 14, 16:04:19 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 14, 15:55:36 GMT+0
Investigating - We are currently investigating this incident..

Duration: 12 hours and 50 minutes

Affected Components:

Mar 4, 06:42:28 GMT+0
Resolved - This incident has been resolved..

Mar 4, 01:10:04 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 3, 17:52:16 GMT+0
Investigating - Due to an issue with our upstream weather provider, METARs may currently be out of date or unavailable. We're working on identifying the exact issue and will update this incident once we know more..

Duration: 1 hour and 44 minutes

Affected Components: , , , , , , , , , , , , , , , , , , ,

Nov 30, 18:02:45 GMT+0
Resolved - This incident has been closed. Our team is still working in the background to evaluate what occurred. We will continue to monitor through the day. .

Nov 30, 16:52:56 GMT+0
Monitoring - Service has been restored at this time. We will continue to monitor before declaring this incident resolved. .

Nov 30, 16:32:27 GMT+0
Identified - We are seeing things begin to stabilize at this time. Our team is still working on full restoration and will continue to update this thread. .

Nov 30, 16:18:29 GMT+0
Investigating - We are currently investigating this incident..

Duration: 52 minutes

Affected Components: , , , ,

Nov 25, 15:52:22 GMT+0
Completed - Maintenance has concluded..

Nov 25, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 25, 15:00:00 GMT+0
Identified - On Monday, November 25th, at 1500z, the AFV server cluster will be unavailable due to a server consolidation. Estimated downtime will be approximately 30 minutes..

Duration: 2 hours and 17 minutes

Affected Components:

Oct 11, 02:15:39 GMT+0
Resolved - This incident has been resolved..

Oct 11, 02:11:13 GMT+0
Monitoring - We have implemented a fix and are currently monitoring the result..

Oct 10, 23:59:00 GMT+0
Identified - Due to a vendor outage, new user registration is temporarily unavailable. We're working on mitigating the issue..

Duration: 3 minutes

Affected Components:

Oct 4, 04:03:04 GMT+0
Completed - AFV Voice Restart is now complete..

Oct 4, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 4, 04:00:00 GMT+0
Identified - The AFV Voice server will be rebooted. Total down time will be less than one minute. Clients will need to reconnect to the voice server after the reboot is complete..

Duration: 30 minutes

Affected Components: , , , , , , , , , ,

Jul 23, 04:30:05 GMT+0
Completed - Maintenance has concluded..

Jul 23, 04:00:30 GMT+0
Identified - Maintenance is now in progress..

Jul 23, 04:00:00 GMT+0
Identified - Kubernetes upgrades will occur on July 23 at 04:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade..

Duration: 22 minutes

Affected Components: , , , , , , , , , ,

Jul 17, 04:22:50 GMT+0
Completed - Maintenance has concluded..

Jul 17, 04:00:58 GMT+0
Identified - Maintenance is now in progress..

Jul 17, 04:00:00 GMT+0
Identified - Kubernetes upgrades will occur on July 16 at 04:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade..

Duration: 7 hours and 52 minutes

Affected Components: , , , , , , , , ,

May 22, 18:21:40 GMT+0
Resolved - This incident has been resolved..

May 22, 13:50:59 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 22, 10:30:00 GMT+0
Investigating - We are aware of issues affecting ATC and pilot client connections to the FSD servers. We're investigating the root cause..

Duration: 1 day and 45 minutes

Affected Components:

May 1, 21:13:04 GMT+0
Resolved - This incident has been resolved..

Apr 30, 22:35:26 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 30, 21:07:35 GMT+0
Investigating - We are currently investigating this incident..

Apr 30, 20:40:30 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 30, 20:28:04 GMT+0
Investigating - We are currently investigating an incident with AFV connection drops..

Duration: 21 minutes

Affected Components: , , , ,

Apr 30, 14:21:09 GMT+0
Completed - This upgrade is now complete..

Apr 30, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 30, 14:00:00 GMT+0
Identified - The AFV Server Cluster is being upgraded. AFV will sustain a full outage during this time..

Duration: 11 minutes

Affected Components:

Apr 8, 02:11:12 GMT+0
Completed - The migration has completed..

Apr 8, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 8, 02:00:00 GMT+0
Identified - The AFV bot server needs to be migrated to a new host for Upstream Provider Reasons. .

Duration: 11 minutes

Affected Components: , , ,

Apr 2, 13:41:19 GMT+0
Completed - This is now complete..

Apr 2, 13:30:01 GMT+0
Identified - Maintenance is now in progress.

Apr 2, 13:30:00 GMT+0
Identified - The US FSD Hub server will be rebooted for an OS Upgrade.

Duration: 1 day and 2 hours

Affected Components:

Mar 17, 17:01:00 GMT+0
Resolved - This incident has been resolved..

Mar 17, 16:35:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 days, 1 hour and 35 minutes

Affected Components: , , , , , , , , , ,

Mar 6, 18:28:34 GMT+0
Resolved - This incident has been resolved..

Mar 6, 18:01:32 GMT+0
Monitoring - All services are back to full functionality. We'll continue monitoring for a bit to ensure stability..

Mar 6, 17:56:20 GMT+0
Identified - We've identified the issue and have verified most services are back to full functionality. We're still working on resolving issues with the METAR service..

Mar 6, 17:39:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 33 minutes

Affected Components:

Mar 3, 00:15:00 GMT+0
Resolved - This incident has been resolved..

Mar 2, 23:53:17 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 2, 23:41:46 GMT+0
Investigating - We are currently investigating this incident..

Duration: 8 hours and 16 minutes

Affected Components: ,

Feb 23, 01:40:22 GMT+0
Resolved - The attack has lifted as of a couple hours ago. We'll continue monitoring for further impact, but at this point the incident is resolved..

Feb 22, 17:24:00 GMT+0
Monitoring - Due to a denial of service attack against our infrastructure, some services may be degraded until the attack ceases. If you encounter errors logging into the network or myVATSIM, please be patient and try again..

Duration: 4 hours and 27 minutes

Affected Components:

Jan 18, 23:55:34 GMT+0
Resolved - This incident has been resolved..

Jan 18, 22:59:57 GMT+0
Monitoring - We've now implemented a fix to get METARs back in order. We'll monitor it for the next while to confirm it's working properly..

Jan 18, 19:29:00 GMT+0
Identified - Our upstream data provider for METARs is returning unexpected data. We're working on implementing a workaround..

Duration: 2 days and 17 hours

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Jan 10, 21:55:00 GMT+0
Resolved - This incident has been resolved..

Duration: 5 minutes

Affected Components: ,

Jan 9, 00:00:00 GMT+0
Completed - Sweatbox-1 will be rebooted tonight at 0000Z .

Duration: 16 minutes

Affected Components:

Jan 4, 03:00:14 GMT+0
Resolved - This incident has been resolved..

Jan 4, 02:50:11 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 4, 02:43:44 GMT+0
Investigating - We are currently investigating this incident..

Duration: 52 minutes

Affected Components: , , , , , , , , , ,

Oct 30, 05:52:27 GMT+0
Completed - Maintenance has concluded..

Oct 30, 05:00:41 GMT+0
Identified - Maintenance is now in progress..

Oct 30, 05:00:00 GMT+0
Identified - Kubernetes upgrades will occur tonight at 05:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade..

Duration: 29 minutes

Affected Components:

Jul 4, 03:28:57 GMT+0
Completed - The support center migration has completed. If you notice any issues, please reach out to the Technology team via [support.vatsim.net](https://support.vatsim.net/open.php)..

Jul 4, 03:01:21 GMT+0
Identified - Maintenance is now in progress. The Support Center will temporarily be unavailable while we complete a migration..

Jul 4, 03:00:00 GMT+0
Identified - We will be performing a migration of support.vatsim.net to a new server at 0300z. The Support Center will be unavailable during that time..

Duration: 1 hour and 27 minutes

Affected Components:

Jun 22, 04:27:09 GMT+0
Completed - The great E-Learning migration has completed. If you notice any issues, please reach out to the Technology team via [support.vatsim.net](https://support.vatsim.net/open.php)..

Jun 22, 03:50:32 GMT+0
Identified - Maintenance is now in progress. Due to the late start, it may last longer than originally estimated..

Jun 22, 03:00:00 GMT+0
Identified - We will be performing a migration of learn.vatsim.net to a new server at 0300z. The E-Learning system will be unavailable during that time..

Duration: 13 minutes

Affected Components: , , , , , ,

May 16, 04:13:00 GMT+0
Completed - Maintenance has completed successfully..

May 16, 04:00:00 GMT+0
Identified - Maintenance is now in progress..

May 16, 04:00:00 GMT+0
Identified - Kubernetes upgrades will occur tonight at 04:00 UTC. Expect web services to be flaky while our compute instances are rotated out..

Duration: 1 hour and 57 minutes

Affected Components:

Mar 8, 03:57:00 GMT+0
Completed - Maintenance has completed successfully..

Mar 8, 02:59:00 GMT+0
Identified - We have started database maintenance. We will keep this page updated if any issues arise..

Mar 8, 02:00:00 GMT+0
Identified - We plan to apply updates to our database cluster during this maintenance window. No user impact is expected during this period of time. We will keep this page updated during the maintenance window..

Duration: 46 minutes

Affected Components:

Jan 31, 02:27:00 GMT+0
Completed - Maintenance has completed successfully..

Jan 31, 01:41:00 GMT+0
Identified - Maintenance is now in progress..