fs2.schmolie.com down (failed storage backplane), restoration completed

The speedups allowed us to complete several very large restores tonight.

The remaining restores are a combination of small restores and complex restores that will take some time to work through. We have a prioritized list and are working through it.

Still on our to-do list is to restore the web interface for dns5.speedingbits.com. We realize that customers have not had access to change/add/delete zones on that system for several days. If you have an urgent DNS entry that needs to be made or changed, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can do it for you manually. If you are able to wait until the web interface is up, please make your entries at that time. We only have very limited time and have to reserve it for urgent issues until everyone's sites are up again.

As we make more progress, we'll post it here.

We're in the process of sorting out the MDH accounts and gathering the web site files from backups.

After we create replacement accounts, upload the files from backups, and recreate all email addresses on the new account, we'll change DNS for each to point to the new server and get in touch with each client to ensure they know how to access the new account.

We've finished restoring the MDH accounts to their new WebPro2 accounts.

We still have several VPS accounts left to restore, most of which we're waiting to hear back from the customer about their choice of control panel before we create the new VPS for them and begin the file restore process.

If you haven't been restored yet and you're not sure why, please login to the billing control panel at https://cbp.speedingbits.com/billing/clientarea.php , review any tickets we've opened for you on the topic and respond with the information requested so we can proceed.

If you've already responded, thank you. You're in the queue to be restored and we'll do so as soon as we can.

We're currently working through some of the more complex restores.

Some of the issues we're working through for certain clients:

• digging up old mysql versions to restore databases from binary files because regular database dumps were not setup by the customer for their databases
• locating 9 year old documentation (from employees who no longer work here) on mysql root passwords or going through mysql root password reset procedures
• working out syntax differences between a mysql dump from a 9 year old version of mysql and a modern version of mysql
• working out compatibility issues with old PHP code because there's a 9 year difference in versions between what they were using and what is current

That's just a small sampling of the tasks we're working through and a big part of why it is taking so long.

For those that never responded to their ticket with a choice of control panel for their e-hwVPS, most of those were recreated today and we'll hopefully be able to get those files uploaded for those clients tonight or tomorrow. The VPS that were recreated today had little to no traffic over the last month, so we question whether they were even in use any more. It was for this reason that they were delayed until after more active clients had been restored.

The process continues...

We're in the process of provisioning the four VPS to replace the four "High Performance VPS COS4x64 2008" services that were affected. These are also getting CPanel/WHM installed before files are restored.

This work is ongoing and will continue through the weekend until complete.

Holy lack of sleep Batman!

We're mostly done with this restore. There are still a few VPS accounts that either aren't being used or are otherwise not noticed by the clients that have not been restored. Prior to the outage, however, none of these had any significant traffic, so it doesn't look like they were being used.

We'll be working to get in touch with those clients.

If you're still having trouble related to this outage (or any kind of trouble, really), please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php .

We are aware that the web interface to dns5.speedingbits.com is still down and that is next on our list to restore. We'll begin that Sunday or Monday (Feb. 11 or 12).

We will continue posting updates on this thread as needed, but they will be less frequent going forward since there are very few still affected and we're probably just talking to the wind now.

We made some progress on the web interface for dns5.speedingbits.com.

We adapted the automation that keeps the zone names synched between the primary and secondary and got that running.

We've also located the backup files and database for the web interface and created a restore plan.

We still need to restore the database to an old version of MySQL, export the database, import/adapt the database for the new version of MySQL/MariaDB, restore the web interface files, adapt the web interface configuration to the new system, test the various functions, debug any PHP version compatibility issues, generate a new valid SSL certificate, and open the firewall so customers can begin using the system again.

Barring unexpected complications, we should have the dns5.speedingbits.com web interface back in service by Wednesday evening.

We are continuing to make progress on restoring the web interface for dns5.speedingbits.com.

After much wrangling, we finally were able to get a working MySQL installation the same version as the original dns5.speedingbits.com was using.

We're now working through restoring our database files to that system from backups so we can take a MySQL dump that can be imported into the newer MySQL version on the new dns5.speedingbits.com.

As indicated in my Feb. 13 post on this thread, there are several additional steps after this that will still need to be done before the web interface is completely restored.

We've successfully restored the mysql database for the dns5.speedingbits.com web interface and are now working on adapting the php code and perl scripts for the web interface so that it will function on this much newer environment.

As indicated in my Feb. 13 post on this thread, there are several additional steps after this that will still need to be done before the web interface is completely restored.

We're still working on the web interface for dns5.speedingbits.com.

Again, while the web interface is down, if you have a need for a new DNS entry or a change to an existing DNS entry on dns5.speedingbits.com, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can make the entry or change manually on your behalf.

Please don't just stew until you scream. Ask for help. It's readily available and willingly offered.

After numerous delays and interruptions from other higher priority projects, we've finally made some progress with the web interface of dns5.speedingbits.com.

We've converted all the configurations to work on the new system. We've setup a working PERL environment and all related modules for the backend automation. We've gotten some of the backend automation working.

Still to do:

• Get all the PHP code working in PHP 7
• Finish getting the remainder of the automation related code reworked for the new system
• SSL certificate provisioning
• Final testing
• Setup daily off-site backups
• Adjust 24/7 monitoring systems

It's still too early to give an ETA for the web interface to be functional again. Getting code working in PHP 7 that was originally written for an older version of PHP can be quick or it could take a great amount of time, depending entirely on the code and how it was written. We won't know how long it will take until we're done, but we're expecting it could be in the range of one work day to several weeks, depending on numerous factors, many not in our control.

This code was written by a third party and the related code project was abandoned by the original author in 2005. The project was setup in such a way that others could not continue the project independently, so it became orphaned. It was at least open source so that we could continue to use and maintain the code we have in spite of no updates from the original author or the community (no project community was allowed to exist by the original author).

Again, while the web interface is down, if you have a need for a new DNS entry or a change to an existing DNS entry on dns5.speedingbits.com, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can make the entry or change manually on your behalf.

Regarding the dns5.speedingbits.com web interface, we've completed the SSL configuration and certificate provisioning.

We're currently working through debugging the old PHP code on the new PHP version. So far, it looks like we have to rewrite all of the mysql_ related functions to support the change in the way PHP accesses MySQL between PHP 4 and PHP 7. There are also a few other error messages we're still tracking down.

With a few more hours of concentrated time and effort, I should be able to complete this major update to the web interface of dns5.speedingbits.com. While rare that I get several hours uninterrupted time, recent changes to technical support duties I previously handled should make this more likely more often.

We've made more progress on the web interface of dns5.speedingbits.com. The database related functions are working correctly now and we've replaced several function calls with their PHP 7 equivalent.

We're probably about 30% done with replacing function calls that no longer exist in PHP 7. We'll be continuing this work tomorrow.

The web interface for dns5.speedingbits.com is nearly complete.

We've finished porting the code to the latest version of PHP, but we're still working to modify the code of the web interface to properly support long DKIM keys in TXT records. This is important for anyone making use of Google email services with their domain since Google requires people to use extremely long DKIM keys that break older DNS systems.

We expect to have this modification complete tomorrow.

The web interface for dns5.speedingbits.com is complete.

Some improvements over the old version:

• Running on fast hardware
• Running on a modern OS with modern PHP, MySQL, and nameserver software
• SSL implemented - no more insecure logins
• Fixed several longstanding bugs
• Now supports Google's gigantic DKIM keys

One important change in how this operates versus previously:
TXT records now require that you put a quotation mark (") at the beginning and end of the text you enter as the value for the TXT record. Previously, this was done for you, but to accommodate special formatting required to handle large DKIM keys, we had to change this.

To use a DKIM key larger than 255 bytes in length, you'll need to break it up into 253 character or less chunks, including spaces, and place quotation marks around each chunk. Insert a space between each quoted chunk of text. The last chunk of text needs a semicolon (;) just before the final quotation mark. For instance:

If you had an account on the old system, it is now active on this new system. Feel free to use it as you did before.

If you encounter any problems with the system, please open a ticket at: https://cbp.speedingbits.com/billing/clientarea.php