DowntownHost Inc
RSS Feed
Latest Updates
Jan
7
Power Failure Miami
Posted by Jorge Catena on 07 January 2015 05:52 PM

All servers located in Miami are down due a power failure on the datacenter.

Servers Affected: 25, 26, 27, 28, 44, 45 and 47.

There's no ETA yet. 


Read more »



Dec
23
[RESCHEDULED] PLANNED NETWORK MAINTENANCE NOTIFICATION (Los Angeles)
Posted by Jorge Catena on 23 December 2014 06:43 PM

Due to unforeseen circumstances the previously scheduled maintenance has been rescheduled. Please see the updated notification below.

###


PLANNED NETWORK MAINTENANCE NOTIFICATION

Location: Los Angeles
Dates: 01/12/2015, 01/13/2015, 01/14/2015
Start: 23:59 PST (GMT-8)
End Time: 04:00 PST (GMT-8)
Expected Downtime: None
Impact: Internet routes may be sub-optimal while bringing routers online

Description:
In order to support our network expansion we are upgrading all border routers on our network and changing our ASN from 29761 to 8100. Due to the redundant nature of our network we do not expect any downtime. Any BGP sessions on our distribution routers will have their ASNs changed during a later maintenance.

If you have any problems with your connectivity after this time, or if you have any questions regarding the maintenance at any point, please contact our support department 

We appreciate your understanding and patience during this maintenance and welcome all feedback.


Read more »



Dec
21
Server 44
Posted by Jorge Catena on 21 December 2014 03:39 PM

We are passing really rough time with the HDD, on 2 weeks we lost about 10 HDDs on different servers, even when we use top noch brands on them, in this case, the problem is on server 44, we got a report of a HDD failing there, we replaced it and while the new HDD was being rebuild on the array, another on the same spun failed, so the server went down, we can't replace that one before the other get rebuilt, so, we tried to start the rebuild process again, but the HDD went offline, we are non-stop working since the friday trying to fix this.

We have no ETA yet, if the issue is not resolved in a couple of hours, we are going to completely rebuild the server from the backups of Dec 18th.

Update 21/12/2014 11:30 AM EST: In approximately 1 hour we will know which path we are going to take. We are trying to clone one of the failed HDD and in one hour we are going to see the progress, if is too slow, we are going to go with the restore.

Update 21/12/2014 1 PM EST: Rebuild is being done at 1% per hour, we will give it 2 more hours to see if it does improve, otherwise, we are going to go with the restore plan.

Update 21/12/2014 3:20 PM EST: The rebuild still was sit at 2% after over 2 hours, we are going to rebuild the whole server and restore it from our backups.

Update 21/12/2014 9:25 PM EST: We are waiting until the DC provides us an external HDD to make a faster backup of the data so the restore can be faster too, since a restore from R1soft of the whole server it-s going to take easily 24 hours.

Update 22/12/2014 10 AM EST: We were able to finish the rebuild of one of the HDD failing, now we replaced the second and it's being rebuilt, everything should be working already fine at the server.


Read more »



Dec
6
Server 25 and 61 Down
Posted by Jorge Catena on 06 December 2014 05:35 PM

Server 61 is again down, when we replaced the faulty BBU of the RAID card the server started to give kernel panic messages, we are actively working on this.

Server 25, the problem is bigger here, on the past week we suffered a problem with the RAID; apparentely came back, however, some data got corrupted, we were trying to fix it, but the best is to rebuild the whole server, before to do that, we are going to move the data to an external HDD to make the restore faster than with R1soft. We are waiting to get the external HDD attached to the server so we can proceed with the migration of the data and then the rebuild.

 

 Update 1: 13:15 EST: Server 61 is already up, External HDD has been attached to server 25.

Update 2: 13:40 EST: We are expecting at least 8 hours in get the data moved to the external HDD, after that, we need to rebuild the whole server, install OS, Cpanel, and then move the data back, we are looking for a downtime of at least 10-12 hours minimum unfortunately.

Update 3: 1 AM EST: We just finished the move of the data to the external HDD; unfortunately it was slower than what we expected, we are now re-installing the OS and Cpanel.

Update 4: 4 AM EST 8/12/2014: After rebuild completely the server, and move the data back, server started to through segmentation fault errors, after investigate, we found that another HDD failed, luckily we changed the other 2 before and the RAID was rebuilt, but we can't continue until the other HDD get replaced since this is a recipe for a complete data loss. People of the DC will replace the HDD at the first hour of the morning, and we will continue after that. We really apreciate your HUGE Patience on this matter.


Read more »



Dec
4
Server 25 Down
Posted by Jorge Catena on 04 December 2014 07:46 PM

Server is down due a problem on the RAID, we are working trying to bring it back ASAP.

Please do not open a ticket nor request chat for updates, we are going to update this announcement with the news that we have. There's no ETA yet.


Read more »




Help Desk Software by Kayako fusion
CONNECT WITH US

If you put our logo in your website, we discount two months of your annual fee!

ABOUT BANNERS