tuxgames.com

tuxgames:

We are not dead

Currently we are experiencing an extended outage caused by the complete failure of British Telecom. This failure is ongoing and we have had so many conflicting reports of what to expect that we can no longer be sure when our service will be back. We are hosting this message on a server that is capable of hosting this message, but not of the entire system for running the company.

Our fault lies in lack of adequately prepared backup measures in the event of a catastrophic failure of service. We will be putting plans in place to remedy this over the coming months.

Just to stress: Our company is still in business, and we have not gone bankrupt, or any of the other rumours that have been flying around

Timeline of events
July 14th 2008: Notified our upstream provider, Eclipse, that we would be moving premesis on the 25th of July. We are told that downtime will be at most 1-2 hours.
July 18th 2008: Eclipse notifies us that to move our service 30 metres further along the street to the new premesis will triple our monthly bandwidth bill
July 18th 2008: We request Eclipse Internet call us to discuss this extreme pricing change
July 24th 2008: Eclipse contacts us and states that the price stands. We cancel our account with Eclipse
July 24th 2008: We contact British Telecom, the only provider who offer an expedited 24 hour installation option on new services. We are told that a new service can be activated within 24 hours of our old service being shut down if we place our order on July 30th
July 30th 2008: Service from Eclipse is suspended. A call to British Telecom places the order for the new service using expedited install, and we are told we should be back online within 24 hours. We are promised a callback to confirm this.
July 31st 2008: No callback from British Telecom. The line is tested throughout the day and no service becomes active
August 1st 208: British Telecom is contacted and they deny having the ability to perform an expedited installation, despite two previous sales representatives offering this service, and our order being placed on this basis alone. I am promised a callback from the originator of the order to explain why we were promised this.
August 4th 2008: No callback has been received from BT, and by this time, we are not willing to rock the boat, as standard install time is 5 working days and this is day 3, and so we expect the service to go live within 48 hours. Our concern is that if we push for the expedited 24 hour install again, it will fail again and we will go back to the back of the 5 day waiting list.
August 6th 2008: The line does not go live as we have been promised. We attmpt to call BT, but the 'final activation time' is also their hometime and so no service representatives are available till tomorrow.
August 7th 2008: We are told that there has been a technical problem closing the order, but that service should resume tomorrow.
August 8th 2008: With no service going live by 3pm, we contact BT again and receive the following explainations at 3.15pm
The order had a technical problem, and should be ready today
BT has had an administrative error and the service should be ready tomorrow
BT has had a critical infrastructure error and the service will be ready on September 10th
At this point, a BT manager promises to investigate this and call us back by 4.30pm.

At 4.45 we contact BT again, and we are told
There is a local fault, we expect them to have it resolved by the end of Monday (the 11th) at the latest
And at 5.30pm we are called back and told
There is a fault between two switching stations, we expect them to have it resolved by the end of September the 4th
August 8th 2008: We investigate alternative suppliers and are pretty much limited to BT, as other services simply run on top of the BT cable, or do not offer the services we need.
August 8th 2008: BT promise to contact us by 12:15pm on Monday (11th) and give us an update.
Our current situation is that we are implimenting a plan that would effectively be our emergency failover plan, which will result in full service being resumed, however as we are not prepared for this entire translation of the entire company architecture from one site to another, it will take some time. In the meantime we hope that BT, a multi-billion pound company, can find SOMEONE competant in their entire organisation and organise replacement of whatever damaged part needs replacing, as I am fairly sure that they can afford to do so if they chose to.

We hope that you, our valued customers, will understand that as a small company reliant on a big company, we are doing our best to solve these problems, but we are to the greater extent just subject to the whims of the larger company. All of our customers are important to us, and we hope you will keep checking here and stick with us when we finally have this sorted out.