VATSIM - Notice history

All systems operational

Notice history

View current status

Oct 2023

OCT30
Kubernetes cluster upgrades
CompletedMaintenance52 minutes
  • Completed
    October 30, 2023 at 5:52 AMUTC
    Completed
    October 30, 2023 at 5:52 AMUTC

    Maintenance has concluded.

  • In progress
    October 30, 2023 at 5:00 AMUTC
    In progress
    October 30, 2023 at 5:00 AMUTC

    Maintenance is now in progress.

  • Planned
    October 30, 2023 at 5:00 AMUTC
    Planned
    October 30, 2023 at 5:00 AMUTC

    Kubernetes upgrades will occur tonight at 05:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade.

FSD netsplit due to upstream network issues
ResolvedMajor outage8 hours 53 minutes
  • Resolved
    UTC
    Resolved

    Our upstream provider has resolved the issue.

    We have enabled traffic to USA West.

  • Update
    UTC
    Update

    We have moved the Dataserver to another region and cordoned USA-WEST. The network is operating at reduced capacity until we have confirmation that the upstream issue is resolved.

  • Update
    UTC
    Update

    Our compute provider has acknowledged the issue, for updates on their end please see here: https://status.digitalocean.com/incidents/gp9bzm1hnlk4

  • Identified
    UTC
    Identified

    USA-WEST FSD is unable to communicate with its hub due to networking issues at our compute provider. Various systems are impacted.

Forum not loading for majority of users
ResolvedMajor outage2 hours 52 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We've now implemented a fix and will be monitoring for a while longer.

  • Investigating
    UTC
    Investigating

    Requests to forum.vatsim.net are returning the error "no healthy upstream" for the majority of our users. We are currently investigating this incident.

USA METARs out of date
ResolvedDegraded performance36 minutes
  • Resolved
    UTC
    Resolved

    We've updated our services to use the new AWC APIs and are now seeing recent USA METARs populate across the network. This incident is now resolved.

  • Identified
    UTC
    Identified

    USA METARs are currently out of date due to changes at AviationWeather.gov. We’re working on a fix and will update this incident once resolved.

Network Connectivity Issues
ResolvedDegraded performance1 hour 37 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We are seeing recovery of all connections at this time. We believe this was related to an incident at an upstream provider..

  • Update
    UTC
    Update

    We are continuing to investigate this incident. We are also aware one of our upstream providers is having a network issue, which may be contributing to our connectivity problems. Updates will follow.

  • Investigating
    UTC
    Investigating

    We are currently investigating potential issues with VATSIM network connectivity. Please stay tuned for updates.

Sep 2023

USA METARs out of date due to AWC outage
ResolvedDegraded performance6 hours 1 minute
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Identified
    UTC
    Identified

    METARs for airports in the United States are currently out-of-date due to issues with the Aviation Weather Center. We're monitoring the upstream issue and will close this incident once we have observed recovery.

FSD USA-WEST Network Connectivity
ResolvedDegraded performance4 hours 59 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved. FSD USA-WEST and Automatic Server Selection are fully operational.

  • Monitoring
    UTC
    Monitoring

    We plan to bring US-WEST and Automatic Server Selection back to full service shortly. Network connectivity issues with one of our providers has been resolved.

  • Update
    UTC
    Update

    We are continuing to keep FSD USA-WEST and one server supporting Automatic Server Selection out of service. All other systems remain fully operational.

  • Update
    UTC
    Update

    FSD USA-WEST and a server supporting Automatic Server Selection have been disconnected from the network. This removes impact from network operations. Once connectivity is restored we will bring them back into service. At this time everything else is operating properly. New connections will be sent to servers not impacted and existing have been migrated to others.

    When those servers were in service, users may have occasionally been unable to connect to FSD or experienced netsplits.

  • Update
    UTC
    Update

    FSD USA-WEST has been taken out of service while connectivity is restored.

  • Identified
    UTC
    Identified

    One of our upstream providers is experiencing networking problems in their SFO region. This is currently affecting FSD's USA-WEST server.

Aug 2023

METAR Service Errors
ResolvedMajor outage2 hours 15 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Update
    UTC
    Update

    A temporary fix has been put in place. We continue to implement a permanent solution.

  • Identified
    UTC
    Identified

    Currently https://metar.vatsim.net is returning 500 errors. We have identified the problem and are working to resolve it.

Networking issues causing Datafeed corruption and Slurper outages
ResolvedDegraded performance7 hours 37 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We've restarted more of our systems and are seeing recovery. Please reach out if you're still seeing issues.

  • Update
    UTC
    Update

    We are currently seeing network connectivity issues at our provider's datacenter in Frankfurt. Pilots and controllers connected to Germany 1 & 2 may occasionally be timed out by the AFV Slurper and lose audio connection.

    We've also observed occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data.

  • Identified
    UTC
    Identified

    While the issues with Slurper have resolved, we're still seeing occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data.

  • Monitoring
    UTC
    Monitoring

    We're starting to see recovery at this time. The Datafeed is back to its normal state and AFV connections are no longer being timed out.
    We'll keep monitoring this incident in case any further issues come up.

  • Identified
    UTC
    Identified

    Our compute provider is experiencing networking issues that are affecting our Datafeed and Slurper systems.

    VAT-Spy, Simaware, and other consumers of the Datafeed may occasionally receive incorrect or corrupt data.
    Pilots and controllers may occasionally become disconnected from Audio for VATSIM.

    Thank you for your patience while we work towards a resolution with our vendors.

Networking issues affecting FSD
ResolvedDegraded performance5 hours 8 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved. Our compute provider has patched their networking systems across all regions and has observed full recovery.

  • Monitoring
    UTC
    Monitoring

    Our compute provider has deployed a fix to all regions. Recovery may still be ongoing as systems return back to normal.
    We'll continue monitoring this issue in case it crops back up.

  • Identified
    UTC
    Identified

    Our compute provider for FSD and other services is currently experiencing issues globally across their network.
    Users may occasionally be unable to connect to FSD or experience netsplits.
    We will update this incident once we have confirmation that the networking issues have been resolved.

Previous

Aug 2023 to Oct 2023

Next