The Biggest & Best Portal to the Professional Property, Workplace and Built Environment Community

Sunday, 25 February

Hacked - Cut Cables Disconnects UK's Largest Data Centre

One of the largest data hosting and cloud solutions providers, UK Fast's promise of '100% Uptime Guaranteed' was shattered week ending December 15 when a construction worker hacked through cables and disconnected the world.

'We know how important it is that your business is always online, which is why we (UK Fast) have invested in fully-owned Tier 3 standard data centres, enterprise-grade infrastructure and our own super-fast network, giving us guaranteed control of the end-to-end process' - unless of course a pick axe wielding construction worker cuts all the cables...and bang goes that theory.

UK Fast is very aware of its PUE - Power Usage Effectiveness - it is one of its major selling points. Unfortunately, if reports are to be believed, the company, one of the largest data hosting and cloud solutions providers in the UK and a major international player, were not so aware of back-up power systems.

 

Data outage

What UK Fast calls a data outage, customers call something else not entirely polite or publishable. Although UK Fast would have it that most customers were happy with the firm's responsiveness, social media tells a different story with UK Fast being lambasted.

UK Fast operates out of plush headquarter offices in Manchester. Thankfully they are plush as some systems folk (up to forty of them) worked 25 hour shifts to bring services back on stream. A complete loss of power was further complicated by  not being able to reboot.

 

Lawrence Jones MBE

UK Fast owner Lawrence Jones is a bit of a tech mogul, owning a techy magazine and events company BusinessCloud and Secarma the cyber security company. He also owns a Swiss ski hotel and an office fit-out business.

He told reporters that most software was up and running with minutes of the power failure but a VMware auto deployer had become corrupted and needing fixing.

Some hosted sites were down for the full 25 hours of the fix.

Jones also explained that two of its power supplies were cut by workmen (it is unlikely that this level of damage was a solitary pick axe and more likely a major digger). He then cited a back-up generator taking time to kick-in - only a few moments but enough to cause a major headache.

 

Protection

A data centre of this size really should have several power options and an uninterruptible system.

Stephen Peal, a director of Yorkshire based uninterruptible power supplies (UPS) distributor and maintenance company PPS Power said: "“This isn’t a site PPSPower support so I cannot comment on what is in place to ensure power continuity but any type of datacentre should have a UPS system in place to ensure there is no drop in power at all. A generator is only there to keep the UPS batteries charged and therefore support the site for much longer, indefinitely if the fuel tank is kept topped up. Further resilience should be installed on a site like this with UPS backing up UPS this is called N+1. Recently we installed an N+1+1 UPS system in a datacentre smaller than what I assume the UKFast site is.”

In a somewhat contradictory message (in that he claimed already to have such a system in place), Jones, has said UK Fast will be investing in a failover system to ensure a power outage will not happen again. He is promising a full report to customers - especially the 100 or so who were affected for the longest time.

An initial statement from UK Fast, which published a timeline throughout the problem starting on December 12 (updated hourly during the incident read: 'At 10:28 GMT our MANOC 5, 6 and 7 facilities were impacted by instability on the incoming mains power as a result of a civil contractor passing a spike through the main feed. This was work being carried at 0.75km away (not at a UK Fast faility) on the path to our onsite transformer. The UPS system supported the load for its designed time and the generators started. Physical damage to the power cable meant service to the site was unstable as a result, the generator sets failed to synchronize and take over service.

 

@Lawrence_Jones

Jones also took to Twitter to thank his staff. 'Huge thank you to the whole team yesterday for your commitment to our customers. The people in this picture enjoying a welcomed breakfast are some of the volunteers who worked a 25 hour shift

Article written by Brian Shillibeer

Share



Related Articles

Power to the People

The majority of PPSPower’s customers are very satisfied with the company’s services and would recommend the business, according to results released in an...

 Read Full Article
1+1 Makes A Sharp Project Sharper

PPSPower has provided a slick N+1+1 UPS solution for the Sharp Project, an unusual office space recently acquired by Manchester City Council. The space previously...

 Read Full Article
Bat Men Go Robbin'

SPIE has 'knicked' the contract for the Heathrow Terminal 5 central battery systems upgrade from under the noses of all other contenders. The deal will involve...

 Read Full Article
Pupils Help Their Schools Prepare For Flooding

Children in the North East are helping their schools prepare for flooding in a series of 'Flood Weeks' run by the Environment...

 Read Full Article
London City Airport Bomb Update

The latest update as of 10:15pm on Monday February 12 is that a WW2 bomb found in the River Thames near London City Airport was due to be removed from a secondary...

 Read Full Article
Countering Terrorism The Hard Way

A new type of concrete, which could significantly lower the number of deaths during bomb blasts, earthquakes and other disasters, is being developed by an expert from...

 Read Full Article
Major Fires Ring In The New Year

New Year's Eve saw a massive fire at a multi-storey car park at Kings Dock in the centre of Liverpool - up to 1,300 cars were destroyed, operator ACC Liverpool have...

 Read Full Article
Body Found In Canary Wharf Trench

The Metropolitan Police, the London Ambulance Service (LAS), the London Fire Brigade and the Canary Wharf Group have all confirmed that construction workers have...

 Read Full Article
Sunday Bowling Turns To Hostage Siege

On Tuesday October 4, David Clarke, aged 53, of Nuneaton, was charged with two counts of false imprisonment, one count of criminal damage, two counts of possession of a...

 Read Full Article
Chief Weather Forecaster's Assessment

Between 3:00pm and 10:00pm on Mon October 16 a spell of very windy weather is expected over parts of the UK - Northern Ireland and the north of Scotland are likely to be...

 Read Full Article