US Node 3 - Unexpected Downtime (Resolved)
  • Priority - Critical
  • Affecting Server - Game US 3 | Chicago
  • Node is offline again. We're trying to get this rebooted and looking in to the cause. We're doing one more normal reboot before looking into it and see if there's any issues with hardware. We'll update you once we have additional information.

     

    EDIT 10:06 pm: This was rebooted from KVM 40 minutes ago but the server is still not responding to pings, and no ssh, since rebooted. I've asked data center technicians to take a look at the display and network connectivity, and a network technician is now checking on the issue. If they don't find an issue with the network we'll be booting it into rescue mode. 

    EDIT 10:19 pm: The node can not find a bootable OS. We are now going to boot it into a rescue iso. We'll attempt to make backups of your data to an offsite storage, if possible, and reinstall the node.

    EDIT 11:50 pm: During the process of trying to boot the server into the rescue CD environment, the server locks up. While troubleshooting why the server was locking up, the technicians discovered that the PSU had gone bad and had caused the power to the SSDs to short out. As a result the SSDs are not detected at all, and when they are connected they start to overheat due to the short on the power circuitry of the SSD. Data is completely non-recoverable. The failed SSDs have been replaced, and we are looking in to restoring data from our backups.

  • Date - 16-06-2020 19:40 - 08-08-2020 20:31
  • Last Updated - 16-06-2020 23:54
US Node 3 - Unexpected Downtime (Resolved)
  • Priority - Critical
  • Affecting Server - Game US 3 | Chicago
  • Node US 3 has unexpectedly gone offline. We're rebooting this node and it should be fixed shortly.

     

    EDIT 11:42 am: The panel should now be able to connect to this node, please wait a while and refresh the page if it doesn't. Only half the server IPs this node work at the moment. We're getting the other half able to ping in 15 minutes.

    EDIT: 12:00 pm: Node is back online and all servers in these location are available.

  • Date - 16-06-2020 08:44 - 16-06-2020 12:12
  • Last Updated - 16-06-2020 12:17
DE Node 4 - Hardware Maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - Game FI 1 | Helsinki
  • We are upgrading our hardware on this node.

    This affects the following server IDs:

    f2c4e348, 9e429e38, 4f8f2c61, 307d1f99, f3309f40, 5d53a330, deddad58, 437a8d32, 3cf9e887, aa8e0bb6, bd4a0c68, 148f1bf0, f66d9a23, a573c21a, 90f4ee4a, 97f8c28c, 7eaa807a, 54e7ad19 and 0081d8c5

    You can find your server ID at the end of the URL your console is on!

    Downtime will be up to 1 hour. We'll start your server when the maintenance is completed.

  • Date - 25-05-2020 22:00 - 16-06-2020 11:28
  • Last Updated - 25-05-2020 22:30
DDoS Attack (Resolved)
  • Priority - High
  • Affecting Server - Game US 3 | Chicago
  • A DDoS attack on this node was successfully mitigated. We'll update this with more information, should any other issues arise.

  • Date - 09-05-2020 11:45 - 09-05-2020 12:00
  • Last Updated - 09-05-2020 12:07
DE Node 3 - Unexpected Downtime (Resolved)
  • Priority - Critical
  • Affecting Server - Game DE 3 | Falkenstein
  • DE 3 Düsseldorf node has unexpectedly gone offline, and come back after manual restart. We will continue to investigate why this occurs.

  • Date - 28-04-2020 04:35 - 28-04-2020 04:40
  • Last Updated - 30-04-2020 01:23
DE Node 3 - Unexpected Restart (Resolved)
  • Priority - High
  • Affecting Other - Game DE 3 | Düsseldorf
  • DE 3 Düsseldorf node has unexpectedly gone offline. We're investigating the cause of this.

    EDIT 2:34 PM CDT: This node unexpectedly crashed and restarted because of issues with SWAP. We apologize for this issue. If your server is offline, you can restart it.

  • Date - 20-04-2020 14:25 - 20-04-2020 14:29
  • Last Updated - 20-04-2020 14:35
US Node 1 - Unexpected Downtime (Resolved)
  • Priority - Critical
  • Affecting Server - Game US 1 | Dallas
  • There's an issue with our provider on US 1 Dallas IP. The problem is a bandwidth overage. We're looking into it

  • Date - 09-04-2020 07:41 - 11-04-2020 12:00
  • Last Updated - 11-04-2020 21:29
DDoS - CA 1 (OVH) (Resolved)
  • Priority - Critical
  • We are currently working to mitigate a DDoS attack against Beauharnois Canada node CA1

    EDIT 11:49 AM CDT: It seems this server was put into rescue mode for some reason. Looking in to the cause. This server might be offline for a while.

    EDIT: 12:25 PM CDT: The DDoS attack on CA 1 was successfully mitigated and the node reconnected. Your servers are back online.

    They put this server into recovery mode because the provider claimed the server was hacked, but we know that it was just a really very strong DDoS attack. We will continue to investigate this, and let you know, if we find any logged information that contradicts this information.

    Please be assured that we take security very seriously.

  • Date - 08-04-2020 11:44 - 08-04-2020 12:25
  • Last Updated - 08-04-2020 13:42
Maintenance - Security Update (Resolved)
  • Priority - Critical
  • Affecting System - Game US 1, US 2, DE 1 and DE 2
  • In about 15 minutes from now, we are performing critical maintenance on US 1 (Dallas), US 2 (Kansas City), DE 1 (Frankfurt) and DE 2 (Frankfurt), to update the daemon software.

    The following server IDs will be affected

    Dallas

    66ec08d8, 1e0e18b5, 13fbbb9d, 33a39602, 8c5c1e8f, a57be5a3, cecb012d, 43a44c59, 206c397f, 9d22550a, 315c13e7, 64da4a50, c0464bee, b25f63ad and 61c448a7

     

    Kansas City

    28ec3276

     

    Frankfurt

    23798440, 30c70b49, 58d852dc, e60aedf3, feb7d223, 2f87da52, cbab0878, f2c4e348, c04b6e46, 0b3742f2, 3a9c6d19, 17dcc991, 4c66671d, 8d44d7b4, b0ffacac, 81c6c5ce, 9c142d33, 51fb78fa, 1d14d302, 550cf2e1 and 7538727a

     

    Your servers will be safely stopped. When the maintenance is completed (about 3 to 5 min), they will be restarted.

  • Date - 30-03-2020 23:44 - 31-03-2020 00:10
  • Last Updated - 31-03-2020 01:46
Unexpected Downtime on Germany Nodes (2) (Resolved)
  • Priority - Critical
  • Affecting Server - Game DE 2 | Falkenstein
  • Our Germany Nodes are both currently offline for unknown reasons. We will provide updates once we have contacted the appropriate people and resolved the issue.

    EDIT 10:10 AM CDT: The nodes are back online, and you can start your server now. We are responsible for this issue...it was a billing error on our part. We sincerely apologize for this downtime.

  • Date - 30-03-2020 06:10 - 30-03-2020 10:10
  • Last Updated - 30-03-2020 10:13
Unexpected Downtime on Germany Nodes (1) (Resolved)
  • Priority - Critical
  • Affecting Server - Game DE 1 | Falkenstein
  • Our Germany Nodes are both currently offline for unknown reasons. We will provide updates once we have contacted the appropriate people and resolved the issue.

  • Date - 30-03-2020 06:08 - 30-03-2020 10:17
  • Last Updated - 30-03-2020 06:10
Emergency Maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - Game US 2 | Seattle
  • An emergency restart was performed on node US 2 Kansas City node.

    The following server IDs were affected: 28ec3276, 6e0ab8be, 6ac34556, bf38df86, e661e8a4, 228bd1b9, 68649626, 222c150b, c7b27ecd, ab778923, 8bc2b8b6, d60235c8 and 7400eb26

     

    These servers are now restarted.

  • Date - 27-03-2020 15:20 - 27-03-2020 15:20
  • Last Updated - 27-03-2020 15:22
Germany Node 1 - Unexpected Downtime (Resolved)
  • Priority - Critical
  • Affecting Server - Game DE 1 | Falkenstein
  • Our Frankfurt, Germany Node 1 is experiencing some issues and is offline. We will provide updates on the status of this as it changes, while we work to get the node back online.

    EDIT 4:57 AM: The issue is now being looked into by technical team

    EDIT 5:14 AM: Attempts to reboot this node have been unsuccessful. Waiting for further intervention.

    EDIT 12:10 PM: Faulty hardware confirmed after we attempted OS reinstall. We are waiting on response from our provider about getting this fixed.

    EDIT 12:12 PM: It seems this was just fixed now. We are getting everything back online.

    EDIT 12:14 PM: Servers are back online. You can start your mc server(s). We apologize for this downtime.

  • Date - 23-03-2020 04:23 - 23-03-2020 12:15
  • Last Updated - 23-03-2020 12:15
Node Restart (Resolved)
  • Priority - Medium
  • Affecting Server - Game DE 2 | Falkenstein
  • EDIT: Servers are coming back online.

     

    A scheduled restart will occur on Germany, Frankfurt node 2, to upgrade RAM. This will begin 7 minutes from now.

    Affected IP: 49.12.11.72 

    You can check if your server uses this IP in your client portal. If your server is on this node, it will be closed for a few minutes and restarted. https://clients.413.io/clientarea.php?action=services

    We'll let you know when this is completed.

  • Date - 21-03-2020 12:00 - 21-03-2020 12:16
  • Last Updated - 21-03-2020 12:13
Network Issue (Resolved)
  • Priority - Critical
  • Affecting Server - Game US 1 | Dallas
  • Reporting that the Dallas 1 node is back up. There was an unexpected network fault in the server. We restarted the link and it works now. Duration of this downtime was: 5 hours, 13 minutes

  • Date - 25-02-2020 06:52 - 25-02-2020 12:06
  • Last Updated - 25-02-2020 13:00
Munich Servers Offline (Resolved)
  • Priority - Critical
  • Affecting System - Game DE 1 and Game DE 2
  • We are investigating the reason why all 2 of our Munich DE nodes are offline. We'll keep you posted.

    Edit at 7:45 AM CST: So far all we can find out is there might have been an error in the data center causing the nodes to be shut down. Under normal circumstances, after an issue like this they should have restarted on their own, but they didn't, and we're waiting to get information about their status and the reason for the shutdown.

    Edit at 11:45 AM CST: Our provider is turning everything on now. We'll have this resolved shortly.

    Edit at 11:52 AM CST: The network is back on. We're going to restart all the servers so we can use this time to restart and update the node.

    Edit at 11:59 AM CST: The issue has been resolved and all Munich Game servers are back online.

  • Date - 17-02-2020 06:28
  • Last Updated - 17-02-2020 12:00
Maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - Game US 1 | Dallas
  • Hardware maintenance is taking place, started 12:00am CST, 6:00am UTC 

    Affected Nodes: US Dallas 1 

    Resolving occasional I/O lag spikes. Expect up to half an hour of downtime for your server. 

    We'll let you know when this maintenance has completed.

     

    EDIT: We're extending this maintenance by about 30 more minutes. Thanks for your patience.

    EDIT: All maintenance has completed. Servers are back online and faster than ever.

  • Date - 05-02-2020 00:55 - 05-02-2020 02:30
  • Last Updated - 05-02-2020 03:15
Network Issue (Resolved)
  • Priority - Critical
  • Affecting Server - Game US 2 | Seattle
  • Kansas City node is back up. DC staff have looked in to the issue at the server's location and resolved it. Sorry for the downtime.

  • Date - 29-01-2020 11:32 - 29-01-2020 12:44
  • Last Updated - 29-01-2020 13:21
Upstream Provider Issues (Resolved)
  • Priority - High
  • Affecting System - VPS and VPN
  • Our upstream providers are reporting a large bandwidth usage. This seems to be a mistake on their end.

    Currently just our VPN services are down, but our Game servers might go offline too until this is resolved.

    We'll make an update when this is resolved.

    EDIT:  The issue appears to be resolved.

  • Date - 01-01-2020 01:34
  • Last Updated - 31-12-2019 22:45

Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.

Server Name HTTP FTP POP3 PHP Info Server Load Uptime
cPanel (s1) PHP Info
Game DE 1 | Falkenstein PHP Info
Game DE 2 | Falkenstein PHP Info
Game US 1 | Dallas PHP Info
Game US 2 | Seattle PHP Info