DowntownHost Inc
RSS Feed
Latest Updates
Dec
21
Server 44
Posted by Jorge Catena on 21 December 2014 03:39 PM

We are passing really rough time with the HDD, on 2 weeks we lost about 10 HDDs on different servers, even when we use top noch brands on them, in this case, the problem is on server 44, we got a report of a HDD failing there, we replaced it and while the new HDD was being rebuild on the array, another on the same spun failed, so the server went down, we can't replace that one before the other get rebuilt, so, we tried to start the rebuild process again, but the HDD went offline, we are non-stop working since the friday trying to fix this.

We have no ETA yet, if the issue is not resolved in a couple of hours, we are going to completely rebuild the server from the backups of Dec 18th.

Update 21/12/2014 11:30 AM EST: In approximately 1 hour we will know which path we are going to take. We are trying to clone one of the failed HDD and in one hour we are going to see the progress, if is too slow, we are going to go with the restore.

Update 21/12/2014 1 PM EST: Rebuild is being done at 1% per hour, we will give it 2 more hours to see if it does improve, otherwise, we are going to go with the restore plan.

Update 21/12/2014 3:20 PM EST: The rebuild still was sit at 2% after over 2 hours, we are going to rebuild the whole server and restore it from our backups.


Read more »



Dec
6
Server 25 and 61 Down
Posted by Jorge Catena on 06 December 2014 05:35 PM

Server 61 is again down, when we replaced the faulty BBU of the RAID card the server started to give kernel panic messages, we are actively working on this.

Server 25, the problem is bigger here, on the past week we suffered a problem with the RAID; apparentely came back, however, some data got corrupted, we were trying to fix it, but the best is to rebuild the whole server, before to do that, we are going to move the data to an external HDD to make the restore faster than with R1soft. We are waiting to get the external HDD attached to the server so we can proceed with the migration of the data and then the rebuild.

 

 Update 1: 13:15 EST: Server 61 is already up, External HDD has been attached to server 25.

Update 2: 13:40 EST: We are expecting at least 8 hours in get the data moved to the external HDD, after that, we need to rebuild the whole server, install OS, Cpanel, and then move the data back, we are looking for a downtime of at least 10-12 hours minimum unfortunately.

Update 3: 1 AM EST: We just finished the move of the data to the external HDD; unfortunately it was slower than what we expected, we are now re-installing the OS and Cpanel.

Update 4: 4 AM EST 8/12/2014: After rebuild completely the server, and move the data back, server started to through segmentation fault errors, after investigate, we found that another HDD failed, luckily we changed the other 2 before and the RAID was rebuilt, but we can't continue until the other HDD get replaced since this is a recipe for a complete data loss. People of the DC will replace the HDD at the first hour of the morning, and we will continue after that. We really apreciate your HUGE Patience on this matter.


Read more »



Dec
4
Server 25 Down
Posted by Jorge Catena on 04 December 2014 07:46 PM

Server is down due a problem on the RAID, we are working trying to bring it back ASAP.

Please do not open a ticket nor request chat for updates, we are going to update this announcement with the news that we have. There's no ETA yet.


Read more »



Nov
25
Server 61 Down
Posted by Jorge Catena on 25 November 2014 04:18 AM

Seems like the problem of the past week (on where the server were powered off without a proper shutdown by the datacenter), was still affecting Server 61, today the server was rebooted but didn't came back, we are activelly working in order of get this fixed ASAP. We extremelly sorry for all the problems that this is causing to you, but please DO NOT open a help desk ticket nor request chat about this, or asking for updates, we are going to do it here, opening a ticket it's going to slow down us in the whole process.

 

Thanks for your patience and understanding.


Read more »



Nov
14
Servers 29, 59 and 61 down
Posted by Jorge Catena on 14 November 2014 07:07 PM

Hello


Today there was a power failure on the facility on where this servers are located, server 61 didn't came back since then, it's throwing a Kernel Panic error, on where we are working now, Server 59 came back, but now is down again, we are still checking what's the problem, and server 29 didn't came back after a reboot.

We appreciate your patience on this matter, please DO NOT open a help desk ticket or request chat, we are fully aware of the issue and working as fast as possible in order of bring everything back.

 


Read more »




Help Desk Software by Kayako fusion
CONNECT WITH US

If you put our logo in your website, we discount two months of your annual fee!

ABOUT BANNERS