All systems operational

Notice history

Nov 2023

METAR outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    We are aware of an upstream issue with our METAR provider (US NOAA). We will post an update when the issue is resolved.

Oct 2023

Kubernetes cluster upgrades
  • Completed
    October 30, 2023 at 5:52 AM
    Completed
    October 30, 2023 at 5:52 AM

    Maintenance has concluded.

  • In progress
    October 30, 2023 at 5:00 AM
    In progress
    October 30, 2023 at 5:00 AM

    Maintenance is now in progress.

  • Planned
    October 30, 2023 at 5:00 AM
    Planned
    October 30, 2023 at 5:00 AM

    Kubernetes upgrades will occur tonight at 05:00 UTC. We expect occasional impact to our web services and audio connections during the upgrade.

FSD netsplit due to upstream network issues
  • Resolved
    Resolved

    Our upstream provider has resolved the issue.

    We have enabled traffic to USA West.

  • Identified
    Update

    We have moved the Dataserver to another region and cordoned USA-WEST. The network is operating at reduced capacity until we have confirmation that the upstream issue is resolved.

  • Identified
    Update

    Our compute provider has acknowledged the issue, for updates on their end please see here: https://status.digitalocean.com/incidents/gp9bzm1hnlk4

  • Identified
    Identified

    USA-WEST FSD is unable to communicate with its hub due to networking issues at our compute provider. Various systems are impacted.

Forum not loading for majority of users
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We've now implemented a fix and will be monitoring for a while longer.

  • Investigating
    Investigating

    Requests to forum.vatsim.net are returning the error "no healthy upstream" for the majority of our users. We are currently investigating this incident.

USA METARs out of date
  • Resolved
    Resolved

    We've updated our services to use the new AWC APIs and are now seeing recent USA METARs populate across the network. This incident is now resolved.

  • Identified
    Identified

    USA METARs are currently out of date due to changes at AviationWeather.gov. We’re working on a fix and will update this incident once resolved.

Network Connectivity Issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are seeing recovery of all connections at this time. We believe this was related to an incident at an upstream provider..

  • Investigating
    Update

    We are continuing to investigate this incident. We are also aware one of our upstream providers is having a network issue, which may be contributing to our connectivity problems. Updates will follow.

  • Investigating
    Investigating

    We are currently investigating potential issues with VATSIM network connectivity. Please stay tuned for updates.

Sep 2023

USA METARs out of date due to AWC outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    METARs for airports in the United States are currently out-of-date due to issues with the Aviation Weather Center. We're monitoring the upstream issue and will close this incident once we have observed recovery.

FSD USA-WEST Network Connectivity
  • Resolved
    Resolved

    This incident has been resolved. FSD USA-WEST and Automatic Server Selection are fully operational.

  • Monitoring
    Monitoring

    We plan to bring US-WEST and Automatic Server Selection back to full service shortly. Network connectivity issues with one of our providers has been resolved.

  • Identified
    Update

    We are continuing to keep FSD USA-WEST and one server supporting Automatic Server Selection out of service. All other systems remain fully operational.

  • Identified
    Update

    FSD USA-WEST and a server supporting Automatic Server Selection have been disconnected from the network. This removes impact from network operations. Once connectivity is restored we will bring them back into service. At this time everything else is operating properly. New connections will be sent to servers not impacted and existing have been migrated to others.

    When those servers were in service, users may have occasionally been unable to connect to FSD or experienced netsplits.

  • Identified
    Update

    FSD USA-WEST has been taken out of service while connectivity is restored.

  • Identified
    Identified

    One of our upstream providers is experiencing networking problems in their SFO region. This is currently affecting FSD's USA-WEST server.

Sep 2023 to Nov 2023