DowntownHost Inc
RSS Feed
Latest Updates
Aug
20
Server 28 down
Posted by Jorge Catena on 20 August 2014 07:09 PM

We found the server unresponsive, after investigate a RAID hardware degradation has been found, we are working in order of bring it back as soon as possible without any data loss, we don't have any ETA yet. This post it's going to be updated whenever we get something new. Please do not open ticket nor request chat about it.

Update 1 16:34 PM EST: We lost 3 of the 4 HDD of the RAID; we are trying to recover the data from the RAID, otherwise, we will need to do it from the backup.

Update 2 17:26 PM EST: We have bad news, all the data has been lost, we are rebuilding the server at this moment, and after that we are going to proceed with the restore from our backup.

Update 3 21:11 PM EST: Restore has been started, we are already restoring /home, so far it's on the 1%.

Update 3: 23:38 PM EST: Restore of Home is on 2%, Mysql in 25%.

Update 4: 8:44 AM EST 08/21/2014: Restore of /home is on 23% mysql has been completelly restored.

Update 5 23:34 PM EST 08/21/2014: Home has been completely restored, we had to restore mysql again, that's what's being done, after that, everything should works fine.


Read more »



Jul
30
Mysql Upgrade
Posted by Jorge Catena on 30 July 2014 10:45 PM

We are forced to upgrade all our servers to Mysql 5.5 otherwise we wont be able to keep Cpanel updated (since it does require that version), it shouldn't be any major issues with the softwares functionality, although, on very old versions it may have some incompatibility problems, unfortunately, this is something that we have to do.

This is going to be done on Saturday 2 of August, at noon.


Read more »



Jun
30
Server 58 down
Posted by Ethan Barnes on 30 June 2014 02:37 PM

The server needs to be rebuild and a restore has been done from backups.

Update: Sorry for the lack of updates but we were virtually bombed with help desk tickets, an announcement here would solve that, but we were running behind of everything this time.

Information about what happened, yesterday at 3:40 AM the server got freeze, a tech cold reboot it and this cause that the root and boot partition get corrupted, we were working for many hours trying to fix it instead of rebuild the server, we were able to restore the boot partition but not root which is on where all the contents of the servers are stored. So, we finally took the decision of rebuild the whole server and restore the contents from our backups and that is what we were doing non stop since yesterday.

At this moment all the services are up, but many sites are still unreachables because the content on /home is still being restored, this is done in alphabetical order, at this moment (9:50 AM EST of July 1) it's on H.

We will keep updating this post every time that we get something new or to keep updating the status of the restore every 2 hours.

We deeply apologies for all the troubles that this is causing to you.

Update 2: Most of the accounts are restored at this moment, if your is still not working, please open a help desk ticket.


Read more »



Jun
27
Server 32 Down
Posted by Jorge Catena on 27 June 2014 09:22 PM

Server 32 is down, we are still working on it trying to find out the reason. We are going to update this announcement as soon as we get something to inform. Please do not open a help desk ticket nor request chat asking for more information.

Update 1: 22:18 EST: Things don't look good, we are still working on this, we have the backups ready to a restore if is needed though.

Update 2: 01:10 AM EST: Our technician has brought this device to our workbench where we are trying to recover the RAID array. Once this is completed we will boot to a Live environment and begin data recovery. 

Update 3: 5:04 AM EST: I believe we were able to recover the root partition. However, due to corruptions  of boot, we are creating  create a copy of root for a new installation of OS and then copy your data back in. Currently running file checks on the partition. Upon its completion, will begin with the data extraction.

Update 4: 6:20 AM EST: Backing up data in progress. After the backup is complete, We will try to restore the /boot partition. If it does not work, the backup will be already made and we can proceed as previously suggested.

Update 5: 9:10 AM EST: Backup is still in progress.

Update 6: 10:56 AM EST: This data copy operation is still in progress.

Update 7: 14:05 EST: The backup is still in progress.

Update 8: 13:15 EST: The backup is in about 60%, unfortunately is going slower than what we initially thought, still this is the faster way to bring everything back online.

Update 9: 17:50 EST: Backup is 68%.

UPdate 10: 22:30 EST: Backup is done, we are rebuilding the server and then we will proceed with the restore. We expect to have some sites working in 2-3 hours (NOT ALL THEM)

Update 11: 2:55 AM EST: We are still restoring data on the server.


Read more »



Jun
21
Server 57 Down
Posted by Jorge Catena on 21 June 2014 12:00 AM

RAID card of this server failed, we are working already on this, unfortunately we don't expect a short downtime. We are going to update this announcement as soon as we get something new to update with.

Please, do not open a help desk ticket, nor request chat about the status, we are going to update here whenever we get something to say.

Update 1: Unfortunately news are not good, we expect a downtime of at least 14 Hours. We are waiting for the new RAID card to be replaced.

Update 2: Replacement RAID card should be delivered in the few hours, sorry for the long delay but the time that's taking it's beyond of our hands.

Update 3: We are trying to reach the company who supposed to deliver the package with the RAID card in there, without luck so far

Update 4: Parts finally arrived to the datacenter, techs are working on it already.

Update 5: Server is back up and running, thanks for your patience and understanding


Read more »




Help Desk Software by Kayako fusion
CONNECT WITH US

If you put our logo in your website, we discount two months of your annual fee!

ABOUT BANNERS