As you would have likely observed last night (10th of November 2023) at roughly 02:52AM GMT we experienced an outage affecting several of our Juniper MX routers across our backbone network that caused network wide disruption for roughly 10-15 minutes.

This later led to additional complications on the underlying CEPH storage layer, which required manual intervention and caused additional downtime / delays.

The cause of this issue was the routing sub process on the MX routers crashing at the same time. There were a few sites which run a different version of JunOS that were unaffected by this, however services in London (DDCL, Telehouse and Equinix LD8), Amsterdam (Iron Mountain AMS1), Paris (Interxion PAR5) and Zurich (Interxion ZUR1) were affected. Frankfurt, Edinburgh, and Glasgow were unaffected. Additionally, our USA-based servers were also unaffected.

We’ve since reviewed log files and crash dumps and reached a consensus internally to roll out a JunOS upgrade to all our MX routers over the course of this next week, even to ones that were unaffected by this issue. After the process is complete all our MX series routers will be running the latest JTAC recommend version.

Due to this being a major release upgrade it will incur some downtime whilst we switch to the new version and the cards within the router reload and protocols fire back up. We anticipate the disruption to last up to 15 minutes during this period; however, interruption to the storage layer is possible, which may extend the outage period.

Our upgrade schedule for routers within London is as follows:

Telehouse North 2 (rt0-thn2) - Saturday 18th November, at risk from 01:00 AM till 02:00 AM GMT

Equinix LD8 (rt0-hex) - Saturday 18th November, at risk from 02:30 AM till 03:30 AM GMT

DDCL (rt0-cr) - Saturday 18th November, at risk from 04:00 AM till 05:00 AM GMT

Services affected by location:
Telehouse North 2 – Any customers with LONAP IX services, carrier pickups that we offer remote backhaul services for or leased line customers with a termination in Telehouse.

Equinix LD8 – Any customers with LINX IX services, carrier pickups that we offer remote backhaul for or leased line customers with a termination in Equinix LD8.

DDCL – This is our primary location for bare metal services within London so during this maintenance period any customers with dedicated / bare metal servers located in London will be considered at risk. Any internal connectivity or private VLAN’s between servers should remain operational.

Whilst any form of maintenance scheduled or not can be disruptive, we’d like to take this opportunity to assure our customers that we’re doing all in our power to ensure our services are ones that you can depend on for your business. We hope this shows in how quickly we’re responding to such an outage and taking the required steps to avoid any further unscheduled disruption.

If you have any questions regarding the maintenance, please do not hesitate to reach out to our support team on support@brixly.uk

paradise.hostns.io recovered.

Apologies for the downtime this morning on the UK infrastructure. There was an incredibly rare incident last night with Juniper network switch firmware, which caused network loss for a brief period.

However, this has led to us needing to rebalance some of the CEPH cluster, which causes increased IO, which inturn crashes Litespeed.

We are working through the issues currently, and will have service resumed as soon as possible.

Thanks for your patience.

paradise.hostns.io went down.

uranium.da.hostns.io recovered.

godfather.hostns.io recovered.

bellini.hostns.io recovered.

derby.hostns.io and shandy.hostns.io recovered.

coral.premium.hostns.io recovered.

derby.hostns.io went down.

margarita.hostns.io recovered.

martini.hostns.io recovered.

moonwalk.hostns.io recovered.

shandy.hostns.io went down.

greyhound.hostns.io recovered.

uranium.da.hostns.io went down.

derby.hostns.io recovered.

bellini.hostns.io went down.

derby.hostns.io and moonwalk.hostns.io went down.

coral.premium.hostns.io went down.

martini.hostns.io went down.

godfather.hostns.io went down.

margarita.hostns.io went down.

Internal SMTP (Notifications, Invoices and Support) recovered.

Internal SMTP (Notifications, Invoices and Support) went down.

ramos.hostns.io recovered.

zombie.hostns.io, sunrise.hostns.io, tequila.hostns.io, and 14 other resources recovered.

taurus.premium.hostns.io went down and tonic.hostns.io recovered.

stout.hostns.io went down.

brixly.uk, client.brixly.uk, and support.brixly.uk recovered.

sangria.hostns.io went down.

zombie.hostns.io went down.

diablo.hostns.io recovered.

absinthe.hostns.io, alexander.hostns.io, aviation.hostns.io, and 34 other resources recovered.

margarita.hostns.io recovered.

scorpius.premium.hostns.io and violet.premium.hostns.io went down and acapulco.hostns.io recovered.

sunrise.hostns.io, tonic.hostns.io, and uranium.da.hostns.io went down.

tequila.hostns.io, savoy.hostns.io, and sunset.hostns.io went down.

ramos.hostns.io went down.

shandy.hostns.io, vesper.hostns.io, virgo.premium.hostns.io, and 1 other resource went down and SpamExperts - mx2.spamfiltering.io recovered.

tuxedo.hostns.io and schnapps.hostns.io went down.

SpamExperts - mx2.spamfiltering.io went down.

diablo.hostns.io went down.

capricorn.premium.hostns.io went down.

martini.hostns.io, da-uk2.hostns.io, and pisces.premium.hostns.io went down.

client.brixly.uk, support.brixly.uk, and pinacolada.hostns.io went down.

derby.hostns.io and gin.hostns.io went down.

emerald.da.hostns.io went down.

brixly.uk, moonwalk.hostns.io, coral.premium.hostns.io, and 2 other resources went down.

paradise.hostns.io and libra.premium.hostns.io went down.

bacardi.hostns.io, margarita.hostns.io, mojito.hostns.io, and 2 other resources went down.

champagne.hostns.io went down.

godfather.hostns.io went down.

brandy.hostns.io and amber.premium.hostns.io went down.

alexander.hostns.io and aviation.hostns.io went down.

absinthe.hostns.io, bellini.hostns.io, greyhound.hostns.io, and 3 other resources went down.

da-uk3.hostns.io went down.

acapulco.hostns.io, corsair.hostns.io, malibu.hostns.io, and 4 other resources went down.