VATSIM - Notice history

USA East 2 experiencing major outage

Notice history

Oct 2023

Kubernetes cluster upgrades
  • Completed
    October 30, 2023 at 5:52 AM
    Completed
    October 30, 2023 at 5:52 AM

    Maintenance has concluded.

  • In progress
    October 30, 2023 at 5:00 AM
    In progress
    October 30, 2023 at 5:00 AM

    Maintenance is now in progress.

  • Planned
    October 30, 2023 at 5:00 AM
    Planned
    October 30, 2023 at 5:00 AM

    Kubernetes upgrades will occur tonight at 05:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade.

FSD netsplit due to upstream network issues
  • Resolved
    Resolved

    Our upstream provider has resolved the issue.

    We have enabled traffic to USA West.

  • Update
    Update

    We have moved the Dataserver to another region and cordoned USA-WEST. The network is operating at reduced capacity until we have confirmation that the upstream issue is resolved.

  • Update
    Update

    Our compute provider has acknowledged the issue, for updates on their end please see here: https://status.digitalocean.com/incidents/gp9bzm1hnlk4

  • Identified
    Identified

    USA-WEST FSD is unable to communicate with its hub due to networking issues at our compute provider. Various systems are impacted.

Forum not loading for majority of users
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've now implemented a fix and will be monitoring for a while longer.

  • Investigating
    Investigating

    Requests to forum.vatsim.net are returning the error "no healthy upstream" for the majority of our users. We are currently investigating this incident.

USA METARs out of date
  • Resolved
    Resolved

    We've updated our services to use the new AWC APIs and are now seeing recent USA METARs populate across the network. This incident is now resolved.

  • Identified
    Identified

    USA METARs are currently out of date due to changes at AviationWeather.gov. We’re working on a fix and will update this incident once resolved.

Network Connectivity Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are seeing recovery of all connections at this time. We believe this was related to an incident at an upstream provider..

  • Update
    Update

    We are continuing to investigate this incident. We are also aware one of our upstream providers is having a network issue, which may be contributing to our connectivity problems. Updates will follow.

  • Investigating
    Investigating

    We are currently investigating potential issues with VATSIM network connectivity. Please stay tuned for updates.

Sep 2023

USA METARs out of date due to AWC outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    METARs for airports in the United States are currently out-of-date due to issues with the Aviation Weather Center. We're monitoring the upstream issue and will close this incident once we have observed recovery.

FSD USA-WEST Network Connectivity
  • Resolved
    Resolved

    This incident has been resolved. FSD USA-WEST and Automatic Server Selection are fully operational.

  • Monitoring
    Monitoring

    We plan to bring US-WEST and Automatic Server Selection back to full service shortly. Network connectivity issues with one of our providers has been resolved.

  • Update
    Update

    We are continuing to keep FSD USA-WEST and one server supporting Automatic Server Selection out of service. All other systems remain fully operational.

  • Update
    Update

    FSD USA-WEST and a server supporting Automatic Server Selection have been disconnected from the network. This removes impact from network operations. Once connectivity is restored we will bring them back into service. At this time everything else is operating properly. New connections will be sent to servers not impacted and existing have been migrated to others.

    When those servers were in service, users may have occasionally been unable to connect to FSD or experienced netsplits.

  • Update
    Update

    FSD USA-WEST has been taken out of service while connectivity is restored.

  • Identified
    Identified

    One of our upstream providers is experiencing networking problems in their SFO region. This is currently affecting FSD's USA-WEST server.

Aug 2023

METAR Service Errors
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Update
    Update

    A temporary fix has been put in place. We continue to implement a permanent solution.

  • Identified
    Identified

    Currently https://metar.vatsim.net is returning 500 errors. We have identified the problem and are working to resolve it.

Networking issues causing Datafeed corruption and Slurper outages
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've restarted more of our systems and are seeing recovery. Please reach out if you're still seeing issues.

  • Update
    Update

    We are currently seeing network connectivity issues at our provider's datacenter in Frankfurt. Pilots and controllers connected to Germany 1 & 2 may occasionally be timed out by the AFV Slurper and lose audio connection.

    We've also observed occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data.

  • Identified
    Identified

    While the issues with Slurper have resolved, we're still seeing occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data.

  • Monitoring
    Monitoring

    We're starting to see recovery at this time. The Datafeed is back to its normal state and AFV connections are no longer being timed out.
    We'll keep monitoring this incident in case any further issues come up.

  • Identified
    Identified

    Our compute provider is experiencing networking issues that are affecting our Datafeed and Slurper systems.

    VAT-Spy, Simaware, and other consumers of the Datafeed may occasionally receive incorrect or corrupt data.
    Pilots and controllers may occasionally become disconnected from Audio for VATSIM.

    Thank you for your patience while we work towards a resolution with our vendors.

Networking issues affecting FSD
  • Resolved
    Resolved

    This incident has been resolved. Our compute provider has patched their networking systems across all regions and has observed full recovery.

  • Monitoring
    Monitoring

    Our compute provider has deployed a fix to all regions. Recovery may still be ongoing as systems return back to normal.
    We'll continue monitoring this issue in case it crops back up.

  • Identified
    Identified

    Our compute provider for FSD and other services is currently experiencing issues globally across their network.
    Users may occasionally be unable to connect to FSD or experience netsplits.
    We will update this incident once we have confirmation that the networking issues have been resolved.

Aug 2023 to Oct 2023

Next