DowntownHost Inc
RSS Feed
Latest Updates
Jun
30
Server 58 down
Posted by Ethan Barnes on 30 June 2014 02:37 PM

The server needs to be rebuild and a restore has been done from backups.

Update: Sorry for the lack of updates but we were virtually bombed with help desk tickets, an announcement here would solve that, but we were running behind of everything this time.

Information about what happened, yesterday at 3:40 AM the server got freeze, a tech cold reboot it and this cause that the root and boot partition get corrupted, we were working for many hours trying to fix it instead of rebuild the server, we were able to restore the boot partition but not root which is on where all the contents of the servers are stored. So, we finally took the decision of rebuild the whole server and restore the contents from our backups and that is what we were doing non stop since yesterday.

At this moment all the services are up, but many sites are still unreachables because the content on /home is still being restored, this is done in alphabetical order, at this moment (9:50 AM EST of July 1) it's on H.

We will keep updating this post every time that we get something new or to keep updating the status of the restore every 2 hours.

We deeply apologies for all the troubles that this is causing to you.

Update 2: Most of the accounts are restored at this moment, if your is still not working, please open a help desk ticket.


Read more »



Jun
27
Server 32 Down
Posted by Jorge Catena on 27 June 2014 09:22 PM

Server 32 is down, we are still working on it trying to find out the reason. We are going to update this announcement as soon as we get something to inform. Please do not open a help desk ticket nor request chat asking for more information.

Update 1: 22:18 EST: Things don't look good, we are still working on this, we have the backups ready to a restore if is needed though.

Update 2: 01:10 AM EST: Our technician has brought this device to our workbench where we are trying to recover the RAID array. Once this is completed we will boot to a Live environment and begin data recovery. 

Update 3: 5:04 AM EST: I believe we were able to recover the root partition. However, due to corruptions  of boot, we are creating  create a copy of root for a new installation of OS and then copy your data back in. Currently running file checks on the partition. Upon its completion, will begin with the data extraction.

Update 4: 6:20 AM EST: Backing up data in progress. After the backup is complete, We will try to restore the /boot partition. If it does not work, the backup will be already made and we can proceed as previously suggested.

Update 5: 9:10 AM EST: Backup is still in progress.

Update 6: 10:56 AM EST: This data copy operation is still in progress.

Update 7: 14:05 EST: The backup is still in progress.

Update 8: 13:15 EST: The backup is in about 60%, unfortunately is going slower than what we initially thought, still this is the faster way to bring everything back online.

Update 9: 17:50 EST: Backup is 68%.

UPdate 10: 22:30 EST: Backup is done, we are rebuilding the server and then we will proceed with the restore. We expect to have some sites working in 2-3 hours (NOT ALL THEM)

Update 11: 2:55 AM EST: We are still restoring data on the server.


Read more »



Jun
21
Server 57 Down
Posted by Jorge Catena on 21 June 2014 12:00 AM

RAID card of this server failed, we are working already on this, unfortunately we don't expect a short downtime. We are going to update this announcement as soon as we get something new to update with.

Please, do not open a help desk ticket, nor request chat about the status, we are going to update here whenever we get something to say.

Update 1: Unfortunately news are not good, we expect a downtime of at least 14 Hours. We are waiting for the new RAID card to be replaced.

Update 2: Replacement RAID card should be delivered in the few hours, sorry for the long delay but the time that's taking it's beyond of our hands.

Update 3: We are trying to reach the company who supposed to deliver the package with the RAID card in there, without luck so far

Update 4: Parts finally arrived to the datacenter, techs are working on it already.

Update 5: Server is back up and running, thanks for your patience and understanding


Read more »



May
21
Miami Servers down: DATACENTER MIGRATION
Posted by Ethan Barnes on 21 May 2014 01:36 PM

DATACENTER MIGRATION - SERVICE IMPACTING

Location: Miami, FL
Date: May 20th, 2014
Start Time: 8:00PM ET
End Time: 8:00AM ET

Servers affected are 25,26,27,28,44,45 and 47.

Update 1: We are in the process of maintenance in Miami servers. We will update once this  gets completed.

Update 2: Server is backup and running.


Read more »



Apr
15
Server 54 issues
Posted by Jorge Catena on 15 April 2014 05:06 PM

We are having problems with Server 54, seems a hardware related issue but we are still working on it, you will see your site going up and down while this happen. We appreciate your patience on this matter, please do not open a help desk ticket nor request chat about it, we are fully aware of the problems and working on it.

Update 1 04/14/2014 14:52 PM: We have initiate the rebuild of the array, we need to replace a couple of HDD that failed at the same time, in order of minimize the risk of data loss we need to replace the HDD one by one, this mean, replace one HDD; rebuild array and then replace the second. We don't have an ETA yet to provide, this announcement it's going to be updated as soon as we get something new.

Update 2 04/14/2014 17:54 PM: Not so great news, Raid 10 is broken into two groups, of two drives.
One group:One drive was rebuilding

One drive stopped responding to controller.

Other group:Looks okay
I am currently manually cloning the drive that stop responding to a new drive, ETA 4+ hours
ETA for total rebuild, if successful, 24 Hours.

Update 3 04/15/2014 00:25 AM: HDD 1 has been cloned, we are going to rebuild the raid now and then clone the next. Still ETA is 20 from now if we are lucky.

Update 4 04/15/2014 11:28 AM: Good news, last night the drive that was failing was successfully cloned. (albeit with a few copy errors)
As of this moment, the 3 drives have been replaced, the 3rd drive is rebuilding. After that finishes, we can insert last drive, boot the system, and let it rebuild live.
I believe it should be ready to boot live in 2-3 hours.

Update 5 04/15/2014 15:20: Server is almost done, we expect to have it online in about 20-30 minutes.

Update 6 04/15/2014 16:30: Server is up and running, however, for some reasons it's unreachable from the outside network, we are working on that.

Update 7 04/15/2014 16:34: Server is backup and running.


Read more »




Help Desk Software by Kayako Fusion
CONNECT WITH US

If you put our logo in your website, we discount two months of your annual fee!

ABOUT BANNERS