PDA

View Full Version : Corvette Server HDD Crashed



Tim Harris
28-08-2012, 01:03 PM
Hello,

WebhostUK would like to extend its profound apologies for the inconvenience caused to the users having their websites hosted on Corvette Server. We have had issues with Corvette Server since past few months and this morning, server went into automatic reboot all of a sudden. We tried rebooting server, however, the server went to chkdsk mode which failed as Hard Disk of the server got corrupted and is failing to detect the windows Operating System. We can try to get the server online through unorthodox methods however, it may put client's website data in jeopardy and might take a lot of time to recover as well. So, to get the server online asap, WebhostUK has decided to setup new High-end configured server, restore the data and route the IP address which will fix the issue permanently. The server configuration is Intel Dual Quad-Core Xeon E5420 2.5Ghz, 12GB RAM, 2 x 500 GB SATA HDDs in Hardware RAID 10. We will be migrating the data manually on the server and the process might take 6-8 hours, depending upon the size of data.

Our Datacenter engineers are working on the issue. Once again, please accept our sincere apologies, we will do our best to get the server up and running asap. We appreciate your patience and co-operation meanwhile. Please check this thread for further updates about data migration as we will keep you posted here.

garethkirk
28-08-2012, 03:21 PM
Hello,

Will the new server come back with the same IP address or will I have to update my DNS settings? If so please could you let me know what the new IP address is?

Thanks

Tim Harris
28-08-2012, 04:28 PM
Hello,

Will the new server come back with the same IP address or will I have to update my DNS settings? If so please could you let me know what the new IP address is?

Thanks

Hello,

Yes, the primary IP address will change however, you will not have to update DNS settings. Our technicians will take care of the changes that needs to be done from the server.

Iteforb
28-08-2012, 06:42 PM
Hello,
when the change server is complete?
are 10 hours which is down.

LL

Albert Kent
29-08-2012, 04:10 AM
Hello,
Our admins are working hard on Issue with Data center admin
We are trying to to replair HDD , Now we have configured new server with high configuration and we are trying to restore sites to new server.

Daniel Smith
29-08-2012, 08:12 AM
Hi,

Just an quick update...
We have just completed the Server setup ( installation of OS / service / etc... ) and now our technicians have started the data migration process from the old server to the brand new server. This process may take next 5 to 8 hours approximately may be shorter than that... WebHostUK Technicians are trying to get this task completed as early as possible.

We thank to all our clients and appreciate there patience.
We will keep updated with the status.

Daniel
29-08-2012, 11:39 AM
As well as this being in breach of SLA, WebHostUK have done nothing to communicate this issue effectively to customers. The server has been down for over 24 hours now, and we're no closer to knowing what is going on.

Please can someone provide a detailed update?

Tim Harris
29-08-2012, 12:36 PM
Hello,

A quick update on data migration, we have migrated 80% data from old server to the new server. Our Data Center engineers successfully managed to retrieve the data which was on the old server before the HDD crashed and have restored it on the new server. As it is Plesk control panel, we cannot do an automated data restoration due to which certain process like creating accounts and databases in Plesk control panel has to me done manually. This process will be completed within 3-5 hours.

We appreciate your patience and co-operation while we get the data restored and server up and running.

pottsm6
29-08-2012, 03:31 PM
I have asked WebHostUK several times for a copy of their SLA which we have a right to see by law as a paying customer of theirs. WebHostUK have ignored this request on each occasion and have deflected the issue with an offensive 1 month hosting refund.

I have independently been monitoring the stability of the Corvette server since the beginning of the year as I was seeing several hours of downtime across my sites every month and didnít think the server was fit to be carrying live traffic or live customer websites. Since Jan 1st 2012 I have collected 32hrs of downtime statistics right up until the server died yesterday morning. I believe WebHostUK were in breach of their SLA before the server went down.

Can someone from WebHostUK please make available your companies SLA and qualify the 0.01% downtime you state in your advertising. Also your TAT and MTTR agreements which should be clearly defined in your SLA as per Ofcomís guidelines for UK web hosting providers.

Daniel
29-08-2012, 03:47 PM
I have asked WebHostUK several times for a copy of their SLA which we have a right to see by law as a paying customer of theirs. WebHostUK have ignored this request on each occasion and have deflected the issue with an offensive 1 month hosting refund.

I have independently been monitoring the stability of the Corvette server since the beginning of the year as I was seeing several hours of downtime across my sites every month and didnít think the server was fit to be carrying live traffic or live customer websites. Since Jan 1st 2012 I have collected 32hrs of downtime statistics right up until the server died yesterday morning. I believe WebHostUK were in breach of their SLA before the server went down.

Can someone from WebHostUK please make available your companies SLA and qualify the 0.01% downtime you state in your advertising. Also your TAT and MTTR agreements which should be clearly defined in your SLA as per Ofcomís guidelines for UK web hosting providers.

This coupled with the frankly awful communication has put a very sour taste in my mouth. I expect to be locating elsewhere within the week.

pottsm6
29-08-2012, 04:06 PM
This coupled with the frankly awful communication has put a very sour taste in my mouth. I expect to be locating elsewhere within the week.

I completely agree! Awful communication is actually being too generous. I still have no idea when (or if) i will be able to access any of the data in my customerís databases or get access to any backups so i can restore elsewhere to other hosting providers.

3 of my bigger customers have left me already today after being offline for over 24hrs and I can't blame them. This is just completely unacceptable and I think everyone with hosting plans on this server should be refunded the full years hosting fee which would probably be a fraction of cost should they get taken to court over breach of their SLA.

TQ
29-08-2012, 04:08 PM
Are we going to be able to retrieve any emails lost during the 2 days since the server crashed?

Tim Harris
29-08-2012, 04:27 PM
I have asked WebHostUK several times for a copy of their SLA which we have a right to see by law as a paying customer of theirs. WebHostUK have ignored this request on each occasion and have deflected the issue with an offensive 1 month hosting refund.

I have independently been monitoring the stability of the Corvette server since the beginning of the year as I was seeing several hours of downtime across my sites every month and didnít think the server was fit to be carrying live traffic or live customer websites. Since Jan 1st 2012 I have collected 32hrs of downtime statistics right up until the server died yesterday morning. I believe WebHostUK were in breach of their SLA before the server went down.

Can someone from WebHostUK please make available your companies SLA and qualify the 0.01% downtime you state in your advertising. Also your TAT and MTTR agreements which should be clearly defined in your SLA as per Ofcomís guidelines for UK web hosting providers.

Hello,

We have responded to your ticket #LHF-930818 6 hours back with update about the issue and TOS. You can refer our TOS on the following link and refer Clause no. 9 : http://www.webhost.uk.net/tos.html

We certainly understand your frustration. However, it is a hardware failure issue which can happen at any hosting company as it is an uncontrollable failure. We appreciate your patience and co-operation while we get the data restored and server up and running.

Daniel
29-08-2012, 04:54 PM
"Tim",

I'm sorry but any hosting company worth it's salt will have off server backups that can be restored onto another physical server in hours, not days. I used to administer Plesk servers - it's easy. Restoring from a RAID array is *not* a backup, and you guys are lucky nothing more serious happened.

And would you care to update the rest of us on what is going on? Replying to one support ticket doesn't enlighten the rest of your customer base.

Iteforb
29-08-2012, 07:03 PM
Hello,
now, someone have website online?

that is a very, very bad situation.

LL

Torg
29-08-2012, 07:55 PM
This is going to lose me lot's of customers. Does anyone have any idea when it's going to work again?

The chat support keeps saying soon but that was yesterday and that's not soon!

gareth.kirk
29-08-2012, 08:05 PM
The service currently being experienced by webhosting.uk.net during this outage is nothing short of terrible. My sites have currently been down for over 36 hours. This is not the first time, there have been constant outages for the last year, coupled with extremely poor communication and service.

All the deadlines published on this forum have been and passed but still no sites. I have also been assured that 90% of the service is back up and running, this does not help the remaining 10%, which my site appears to be part of.

When is this problem expected to be resolved once and for all so i can get a copy of my data and move it to a reliable hosting provider?

pottsm6
29-08-2012, 08:12 PM
I have also been assured that 90% of the service is back up and running, this does not help the remaining 10%, which my site appears to be part of.


Gareth,

All (9 of my sites) must be part of the 10% still down as well then as i have had nothing come back up for any of them...

Iteforb
29-08-2012, 08:24 PM
Hi,
also all my 5 website are into 10%?

Only mail address for 1 website to work.

LL

kompound
29-08-2012, 08:24 PM
My site is still down... 1.5 days...

It wouldn't be an issue if you actually told me what's going on but instead I get "the data will be migrated within 4-6 hours" ok fine.. 8 hours later I get "engineers are working hard on it, data will be migrated in 2-3 hours" etc etc, had this 4 times now.

Then the biggest kick in the teeth is now when I have attempted to do live chat with George Shaw he just ignores me or leaves chat, on 4 occasions now.

I really don't understand why you are pushing your customers away and want them to leave?? Basic customer services, always keep the customer up to date and provide a sweetener / compensation to retain the customer. Sure hardware will fail, we understand there will be downtime in a situation like this but sticking your head in the sand is not a solution and ultimately will piss a whole bunch of people off.

If you're going to retain your customers it's going to take more than a measly "1 month free hosting" as quoted by one of your live chat agents, I nearly fell off my chair when she said she was serious... HAHA. WOW £20 refund when I've potentially lost £0,000's in business, sounds like a good deal to me... oh dear Webhost UK, sort it out.

p.s. Ironically, I was raving about you guys to a potential customer a week ago and I was going to order a 2 x VPS this week... Think I'll give that a rethink

Torg
29-08-2012, 08:27 PM
100% of my sites are in this 10%. Terrible.

Torg
29-08-2012, 08:33 PM
Ok, just checked, things seem to be up but ip address changed to 92.48.118.139, I thought the ip address wasn't going to change?

Albert Kent
29-08-2012, 09:09 PM
Unfortunately, complete data could not be retrieved as after restoring the data, we found that both HDD on the server got corrupted. We have managed to recover most of the data. However, for few accounts, we will have to restore old data available on our offline backup server. The process will take next 4-5 hours.

We sincerely apologize for the inconvenience caused to our clients. We appreciate your patience and co-operation while we get the data restored and server up and running.

Abbie
29-08-2012, 09:52 PM
Hello All...

This is Abbie here on behalf of WEBHOSTUK. Well, I would like to start with one simple statement; I am not here to lie or to defend our company. I am here to let everyone know that the situation of Corvette being down for such a prolonged time since yesterday is something that is being worked by the ENTIRE staff of WEBHOSTUK on the highest priority.

We completely and personally understand how it feels when your business website is down for such a prolonged time and your clients fuming at you. Had it been my personal website, I would and will feel the same... Helpless and frustrated..! But, does it really resolve things? Fuming on the company where your websites is hosted, is just going to land your situation nowhere --- > especially when the company staff is doing all it can to restore the things in the best possible way..!

I would like to mention a few things here, which I believe should clear a few doubts and misunderstandings.. Like said earlier, not trying to defend, however misunderstandings lead nowhere except grievances.

Well Daniel mentioned somewhere in the thread above, why did WEBHOSTUK did not maintain offsite backups. We do have offsite backups for Corvette server (INSPITE OF THE FACT WHEN WE DID NOT MENTION OR PUBLISH OR PROMISE IT ON OUR WEBSITE FOR SHARED WEB HOSTING PLANS). Did we? Not at all..! In spite of this, WEBHOSTUK took offsite backups for our shared windows hosting just to be sure that we have our customers to offer something that other web hosting companies do not offer. I would here like to exceptionally promote our company against all those competitors out there in terms of data and backups, do they offer something that they have not promised (in terms of data restore)?

Again, this does not mean, we have the entire backup. Offsite backups were configured on weekly basis which have the last weeks backups available...

Not to forget, our TOS clearly mentions in clause no 5. "Your use of the service is at your sole risk. WebHost.UK.Net is not responsible for files and/or data residing on your account. You agree to take full responsibility for files and data transferred and to maintain all appropriate backup of files and data stored on WebHost.UK.Net servers."

We have always recommended our customers to maintain their own copy of backups. Hardware failures are bound to happen in any web hosting business. When Corvette was setup years before for Windows hosting, it was one of the best hardware servers. Unfortunately, its drive crashed before we could plan its migration to a new configuration server. Yes my statement comes with a confirmation that WEBHOSTUK did have plans to upgrade Corvette to latest powerful hardware servers. However, before we could announce this upgrade which was supposed to be performed in the month of November, Corvette hard drive left us failing and helpless.

We have secondary drive on Corvette which maintained all the latest backups for all accounts. It included your web files, mails, and databases. However, again we are here dealing with Plesk (which is unfortunately the only popular control panels for Microsoft Windows). My friend Daniel said, restore in Plesk is easy.. I wish it was that easy and comfortable when it comes to restoring everything manually...! We have the best Windows guys here employed in Parallels Plesk itself who are working on restore, and still struggling to get it done with the best accuracy. There are plenty of reasons behind this... few of them being, the Windows IIS permissions issue, email migration issue, the database restore issue, which needs to be done manually and not automated with just pack and unpacking the backup file. I am not a technical person, however having the Parallels Windows best technician working in front of my desk who are concentrated so much so that they haven't slept a wink since yesterday makes me believe, we are here dealing with the most awful hdd crash but optimistic and certain to get most of the data (accounts, email, databases) restored…!

Now the question comes, why did this happen, and if so, why did WEBHOSTUK did not do anything to prevent such accidents from happening. Every person hosted with us has a right to question, why? I would like to inform all the customers of WEBHOSTUK, all our Linux and Windows servers are backed up daily, weekly onsite and offsite (in spite of the fact that we have clearly requested in our TOS to have every customer of ours to maintain their own copy of backup). Since Jan 2011, all our servers are configured with RAID10; thus leaving no chance of data loss. However, few of our old servers which do not have higher RAID configurations, are in the process of upgrade and I am sure our other customers (not the one’s from Corvette) have got emails about the upgrade schedules as per our plan. Upgrade procedures not being that simple as they look, as we have to first inform our customers beforehand, and then make sure they do not suffer downtime (in spite of the hardware and IP change - where applicable). Like mentioned, Corvette was one of them on the list which was scheduled to be upgraded in the month of November 2012, while its announcement was supposed to come in mid-September 2012.

Coming back to the current status for Corvette restore, our engineers have confirmed about the restore being done successfully as of now. I am sure, many of our customers might face permission errors with their website, emails and databases, however, I would humbly request you people to submit a ticket with our support desk, since this has been a manual restore and might need a manual fix.

I would like to request from all the customers on Corvette, our sincere apologies and patience while we deal with this dreadful situation. Words have no ends and our efforts of honesty probably could not be described to our valuable customers. Nothing wrong with it, when they are going through the tough situation of their business being down because of the hardware failure our server suffered. However, would just like to be honest about the fact, we all here are WEBHOSTUK are doing all what we can, to make sure your websites and business is back up..! If you could please pass on our humble apologies to your end user clients and website visitors; that would really be one of the biggest things that would happen to all of us. Our only intention here is to assure you that the staff here at WEBHOSTUK is one of the most qualified and best staff, doing their best to bring your websites back to normal. I hope if this post of mine could assure you that we are here to do all that we can, with no intention to harm your business and websites….

Sincerely,
Abbie Clayton

Iteforb
30-08-2012, 06:49 AM
hello,
all beautiful, but at moment all my websites are down and is second day.

LL

pottsm6
30-08-2012, 10:27 AM
Anyone had any success at logging into the sql server dbs yet?

I have now managed to ftp all my websites from the new server to my laptop (so feeling 1% happier), but still can't access the db via server IP or existing dns names?

pottsm6
30-08-2012, 05:28 PM
WEBHOSTUK, please can you give us an update on the situation as its nearly been 24 hrs since the last update on this public forum?

Are you any closer to getting the DNS' switched on / working?
Are you any closer to getting the SQL Server back up and databases restored?
What is the current process you are following for complete restoration and how far into this process are you?
Any idea of when it will be completely restored?

Any information would be great.

Regards,
Matt

Newbury
30-08-2012, 06:51 PM
I guess most people will be wanting to leave webhost.uk now. Any suggestions for a sort of 'class action' to get some compensation from them (eg refund of money paid)?

Iteforb
30-08-2012, 07:08 PM
Hello Newbury,
them offer 1 month for compensation, but i think is few for 3 days with websites down.
write here your suggestion.

LL

Tim Harris
31-08-2012, 10:09 AM
Hello All,

We have restored the data onto new server and most of the domains have started functioning and server is online. As it was a manual migration, few domains might give permission or database error. So, kindly report the issue by email to support@webhost.uk.net so that we can get it fixed immediately. Our Data Center engineers have restored the data available in backup. However, if you have recent backup, you can raise a ticket and we will restore the backup for you.

Regarding compensation, though it is mentioned in our TOS Clause no. 9.C. (Hardware Failure is an uncontrollable disaster for which compensation is not applicable) we have provided compensation. It is clearly mentioned in our TOS Clause no.9.B. that the total credit applicable would never be more than one month's hosting charges paid by the customer at any time and for whatever reason. We certainly understand our client's frustration and personally understand where they are coming from because of the downtime for more than 36 hours and being in your situation we would have been to. But considering the fact, that it was Hardware Failure, an uncontrollable disaster which can occur without warning, we managed to get the server online in the quickest possible way by setting up new high-end configuration raid array server and restoring the backup available with us for customer's data security. We had scheduled migration for Corvette Server in the month of November 2012 on a RAID array server but unfortunately, Server crashed sooner.

WEBHOSTUK would like to thank the clients for keeping their patience and co-operating with us during this dreadful situation. I would just like to assure you that the complete WEBHOSTUK STAFF has given their best to get the websites online on the highest priority, with no intention to damage your websites and business.

I would request clients who are still facing issues to send an email to support@webhost.uk.net and we will get it fixed immediately.

Sincerely,
On Behalf of WebhostUK

pottsm6
31-08-2012, 11:31 AM
Although my websites came back online towards the end of yesterday i still see this server issue as far from resolved.

SQL Server is still not allowing remote connections (from any application i try) and if anyone else in the forum community (that does not work for WebHostUK) can verify this is the case for you as well and not just me i would be grateful.

I have had a long and painful ticket still on-going about enabling the remote connections on SQL Server so i can access my databases from SQL Server Management Studio or Visual Studio or anything else that is not the web based version of Enterprise manager.

Web Host UK are saying that this is all enabled and they have tested it and it is working fine but i dont believe it is.

As far as i am concerned this crash of the server is still not fully restored as i still haven't got the service that i had on monday evening so for me, this has now been nearly 78 hours down!

pottsm6
31-08-2012, 11:48 AM
Hello All,

Regarding compensation, though it is mentioned in our TOS Clause no. 9.C. (Hardware Failure is an uncontrollable disaster for which compensation is not applicable) we have provided compensation. It is clearly mentioned in our TOS Clause no.9.B. that the total credit applicable would never be more than one month's hosting charges paid by the customer at any time and for whatever reason. We certainly understand our client's frustration and personally understand where they are coming from because of the downtime for more than 36 hours and being in your situation we would have been to. But considering the fact, that it was Hardware Failure, an uncontrollable disaster which can occur without warning, we managed to get the server online in the quickest possible way by setting up new high-end configuration raid array server and restoring the backup available with us for customer's data security. We had scheduled migration for Corvette Server in the month of November 2012 on a RAID array server but unfortunately, Server crashed sooner.


Tim,

What should i do about the below tickets which i have raised over the last 2 months that have all resulted in loss of service and which were all accpeted and fixed by WebHostUK? Am i goig to recieve any compensation for these?

07 Jun 2012 1:37 PM (NLT-637946) - 2.5 hours downtime
08 Jun 2012 12:32 PM (SPG-994884) - 4 hours downtime
17 Jul 2012 9:03 AM (BIA-559824) - 1 hour downtime
15 Aug 2012 2:31 PM (FRK-472817) - 3 hours downtime
28 Aug 2012 9:14 AM (QGC-834795) - 72 hours + downtime

Also, can you please let me know how i can view where this compensation has been paid to or refunded?

Thanks

ajitsharma2000
31-08-2012, 01:15 PM
hi,
my site was down for more then two days, still one of my site is down and even on support ticket since 24 hours no body answered, on live chat stopped responding. do webhost uk really stopped thinking about customers. also will anybody answer me that when my site will work proper.

regards
AS

Newbury
04-09-2012, 02:15 PM
My accounts still aren't working after a week. I've lodged a complaint with the Advertising Standards Authority about the '99.99% uptime guarantee'. I suggest others do the same.

In my dictionary, 'guarantee' is defined as: "a formal assurance, esp in writing, that a product, service, etc will meet certain standards or specifications". I also mentioned their front page statement: "Why Choose Us? 100% customer satisfaction" !!