DowntownHost Inc
RSS Feed
Latest Updates
Jan
28
Server 27 Down
Posted by Sebastian Mateo on 28 January 2015 10:43 PM

Something is wrong with the RAID configuration, suddenly two HDD show as down and therefore the server is down, we are working on that.

Update 1: 6:30 PM EST: Server is back, however, still one HDD is failing, we are still working on the server

Update 2 7:45 PM EST 01/29/2015: Unfortunately RAID failed again, 3 HDD failed in less than 24 hours, today we replaced 2 HDD but there was a corruption on the RAID that it does not allow to the server boot, we are looking to mirror the partition to an external HDD to rebuild the server after. This is the option that at this moment look that its going to take, meanwhile we are looking for alternatives.

 Update 3 10 PM EST 01/29/2015: Unfortunately we dont have good news about this, the damage of the RAID is too big to clone it, and on the same day 2 HDD failed on the backup server making it unusable, we are facing some issues with data lost at this moment, we are doing everything that we have in hand in order of recover it.

We are waiting to a new tech to arrive to the Datacenter tomorrow morning who is going to explore new options, unfortunately, is a tech of the datacenter and he start his shift at 7 AM EST.

Update 4 10:10 AM EST 01/30/2015: We are still waiting to get the external HDD attached to the server 27. The rebuild of the backup server is at 31% with an average rate of 3% per hour. There are 50/50 chances of get the backup back. We are a bit more optimistic about the RAID of server 27, like 60/40. We are going to keep updating this announcement as soon as we get something new.

Update 5 12:07 AM EST 01/30/2015: Backup RAID rebuild its at 38%.

Update 6 2:13 PM EST 01/30/2015: External HDD arrived, we are working on that, will post some news about the progress shortly. Backup RAID rebuilt 45%.


Read more »



Jan
7
Power Failure Miami
Posted by Jorge Catena on 07 January 2015 05:52 PM

All servers located in Miami are down due a power failure on the datacenter.

Servers Affected: 25, 26, 27, 28, 44, 45 and 47.

There's no ETA yet. 


Read more »



Dec
23
[RESCHEDULED] PLANNED NETWORK MAINTENANCE NOTIFICATION (Los Angeles)
Posted by Jorge Catena on 23 December 2014 06:43 PM

Due to unforeseen circumstances the previously scheduled maintenance has been rescheduled. Please see the updated notification below.

###


PLANNED NETWORK MAINTENANCE NOTIFICATION

Location: Los Angeles
Dates: 01/12/2015, 01/13/2015, 01/14/2015
Start: 23:59 PST (GMT-8)
End Time: 04:00 PST (GMT-8)
Expected Downtime: None
Impact: Internet routes may be sub-optimal while bringing routers online

Description:
In order to support our network expansion we are upgrading all border routers on our network and changing our ASN from 29761 to 8100. Due to the redundant nature of our network we do not expect any downtime. Any BGP sessions on our distribution routers will have their ASNs changed during a later maintenance.

If you have any problems with your connectivity after this time, or if you have any questions regarding the maintenance at any point, please contact our support department 

We appreciate your understanding and patience during this maintenance and welcome all feedback.


Read more »



Dec
21
Server 44
Posted by Jorge Catena on 21 December 2014 03:39 PM

We are passing really rough time with the HDD, on 2 weeks we lost about 10 HDDs on different servers, even when we use top noch brands on them, in this case, the problem is on server 44, we got a report of a HDD failing there, we replaced it and while the new HDD was being rebuild on the array, another on the same spun failed, so the server went down, we can't replace that one before the other get rebuilt, so, we tried to start the rebuild process again, but the HDD went offline, we are non-stop working since the friday trying to fix this.

We have no ETA yet, if the issue is not resolved in a couple of hours, we are going to completely rebuild the server from the backups of Dec 18th.

Update 21/12/2014 11:30 AM EST: In approximately 1 hour we will know which path we are going to take. We are trying to clone one of the failed HDD and in one hour we are going to see the progress, if is too slow, we are going to go with the restore.

Update 21/12/2014 1 PM EST: Rebuild is being done at 1% per hour, we will give it 2 more hours to see if it does improve, otherwise, we are going to go with the restore plan.

Update 21/12/2014 3:20 PM EST: The rebuild still was sit at 2% after over 2 hours, we are going to rebuild the whole server and restore it from our backups.

Update 21/12/2014 9:25 PM EST: We are waiting until the DC provides us an external HDD to make a faster backup of the data so the restore can be faster too, since a restore from R1soft of the whole server it-s going to take easily 24 hours.

Update 22/12/2014 10 AM EST: We were able to finish the rebuild of one of the HDD failing, now we replaced the second and it's being rebuilt, everything should be working already fine at the server.


Read more »



Dec
6
Server 25 and 61 Down
Posted by Jorge Catena on 06 December 2014 05:35 PM

Server 61 is again down, when we replaced the faulty BBU of the RAID card the server started to give kernel panic messages, we are actively working on this.

Server 25, the problem is bigger here, on the past week we suffered a problem with the RAID; apparentely came back, however, some data got corrupted, we were trying to fix it, but the best is to rebuild the whole server, before to do that, we are going to move the data to an external HDD to make the restore faster than with R1soft. We are waiting to get the external HDD attached to the server so we can proceed with the migration of the data and then the rebuild.

 

 Update 1: 13:15 EST: Server 61 is already up, External HDD has been attached to server 25.

Update 2: 13:40 EST: We are expecting at least 8 hours in get the data moved to the external HDD, after that, we need to rebuild the whole server, install OS, Cpanel, and then move the data back, we are looking for a downtime of at least 10-12 hours minimum unfortunately.

Update 3: 1 AM EST: We just finished the move of the data to the external HDD; unfortunately it was slower than what we expected, we are now re-installing the OS and Cpanel.

Update 4: 4 AM EST 8/12/2014: After rebuild completely the server, and move the data back, server started to through segmentation fault errors, after investigate, we found that another HDD failed, luckily we changed the other 2 before and the RAID was rebuilt, but we can't continue until the other HDD get replaced since this is a recipe for a complete data loss. People of the DC will replace the HDD at the first hour of the morning, and we will continue after that. We really apreciate your HUGE Patience on this matter.


Read more »




Help Desk Software by Kayako fusion
CONNECT WITH US

If you put our logo in your website, we discount two months of your annual fee!

ABOUT BANNERS