VATSIM - Notice history

All systems operational

Notice history

Oct 2023

FSD netsplit due to upstream network issues
  • Resolved
    Resolved

    Our upstream provider has resolved the issue.

    We have enabled traffic to USA West.

  • Identified
    Update

    We have moved the Dataserver to another region and cordoned USA-WEST. The network is operating at reduced capacity until we have confirmation that the upstream issue is resolved.

  • Identified
    Update

    Our compute provider has acknowledged the issue, for updates on their end please see here: https://status.digitalocean.com/incidents/gp9bzm1hnlk4

  • Identified
    Identified

    USA-WEST FSD is unable to communicate with its hub due to networking issues at our compute provider. Various systems are impacted.

Network Connectivity Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are seeing recovery of all connections at this time. We believe this was related to an incident at an upstream provider..

  • Investigating
    Update

    We are continuing to investigate this incident. We are also aware one of our upstream providers is having a network issue, which may be contributing to our connectivity problems. Updates will follow.

  • Investigating
    Investigating

    We are currently investigating potential issues with VATSIM network connectivity. Please stay tuned for updates.

Sep 2023

FSD USA-WEST Network Connectivity
  • Resolved
    Resolved

    This incident has been resolved. FSD USA-WEST and Automatic Server Selection are fully operational.

  • Monitoring
    Monitoring

    We plan to bring US-WEST and Automatic Server Selection back to full service shortly. Network connectivity issues with one of our providers has been resolved.

  • Identified
    Update

    We are continuing to keep FSD USA-WEST and one server supporting Automatic Server Selection out of service. All other systems remain fully operational.

  • Identified
    Update

    FSD USA-WEST and a server supporting Automatic Server Selection have been disconnected from the network. This removes impact from network operations. Once connectivity is restored we will bring them back into service. At this time everything else is operating properly. New connections will be sent to servers not impacted and existing have been migrated to others.

    When those servers were in service, users may have occasionally been unable to connect to FSD or experienced netsplits.

  • Identified
    Update

    FSD USA-WEST has been taken out of service while connectivity is restored.

  • Identified
    Identified

    One of our upstream providers is experiencing networking problems in their SFO region. This is currently affecting FSD's USA-WEST server.

Aug 2023

METAR Service Errors
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Update

    A temporary fix has been put in place. We continue to implement a permanent solution.

  • Identified
    Identified

    Currently https://metar.vatsim.net is returning 500 errors. We have identified the problem and are working to resolve it.

Networking issues causing Datafeed corruption and Slurper outages
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've restarted more of our systems and are seeing recovery. Please reach out if you're still seeing issues.

  • Identified
    Update

    We are currently seeing network connectivity issues at our provider's datacenter in Frankfurt. Pilots and controllers connected to Germany 1 & 2 may occasionally be timed out by the AFV Slurper and lose audio connection.

    We've also observed occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data.

  • Identified
    Identified

    While the issues with Slurper have resolved, we're still seeing occasional corruption of the Datafeed. Applications using the Datafeed (SimAware, VAT-Spy, VATprism, etc) will occasionally not be able to fetch network data.

  • Monitoring
    Monitoring

    We're starting to see recovery at this time. The Datafeed is back to its normal state and AFV connections are no longer being timed out.
    We'll keep monitoring this incident in case any further issues come up.

  • Identified
    Identified

    Our compute provider is experiencing networking issues that are affecting our Datafeed and Slurper systems.

    VAT-Spy, Simaware, and other consumers of the Datafeed may occasionally receive incorrect or corrupt data.
    Pilots and controllers may occasionally become disconnected from Audio for VATSIM.

    Thank you for your patience while we work towards a resolution with our vendors.

Aug 2023 to Oct 2023

Next