Printable Version of Topic
Click here to view this topic in its original format
forum.schmolie.com > Outages and Maintenance > fs2.schmolie.com down (failed storage backplane)


Posted by: andy Feb 4 2018, 04:23 PM
fs2.schmolie.com is currently under heavy load which is slowing down all services running on that machine. The following systems are affected: mdh0.speedingbits.com, multiple resolver and authoritative DNS servers, and 16 VPS accounts from various clients.

We're investigating the cause and hope to resolve it soon.

New information will be posted to this thread as it becomes known.

Posted by: andy Feb 4 2018, 04:32 PM
Remote access to the system does not appear to be functioning, so a technician has been dispatched.

Posted by: andy Feb 4 2018, 05:09 PM
The system appears to be in the middle of replacing a failed hard drive from a hot spare. The system should not be going unresponsive during this process, though, so we're investigating further.

Posted by: andy Feb 4 2018, 05:16 PM
It looks like the replacement hard drive failed in the middle of the replacement and a bug in the RAID controller firmware took the storage offline rather than continuing in degraded mode.

We're power cycling the system, which should bring it back online.

Posted by: andy Feb 4 2018, 05:33 PM
Unfortunately, it gets worse.

Due to another firmware bug in the RAID controller, one of the member disks was marked as a hot spare and the RAID controller began automatically rebuilding onto that disk. It sounds like a fine thing until you realize that it just catastrophically ate a full disk's worth of data, sending it to oblivion, when it presumed the RAID 5 array to be 6 disks instead of 7.

We'll be beginning the restore process soon.

If your account is one of the affected web sites or VPS on this node, please bear with us while we work to bring this system back online from backups.

ETA is 12 to 16 hours to have everything restored.

If you cannot wait that long, please order a new account and you can either restore from your personal backups or instruct us to restore, which we will do as soon as possible.

Posted by: andy Feb 4 2018, 06:22 PM
On closer examination and additional thought, the root cause is likely a failed backplane for the hard drives. We don't have a spare backplane on hand for this type of machine, so we're going to be restoring to other machines.

Please avoid contacting off-hours support about this issue. We're obviously aware of it and working on it as quickly as possible. Screaming at us to work faster won't help.

If you have system which is otherwise unrelated but experiencing difficulties, it may be because of the DNS resolvers that have gone down with this node. If you're experienced and have a VPS, you can change your resolvers in /etc/resolv.conf . Ping each one listed there, remove the ones that are down and save the file.

If you're inexperienced, do not attempt to edit the /etc/resolv.conf on your own and, instead, open a ticket at https://cbp.speedingbits.com/billing/clientarea.php to get assistance.

Posted by: andy Feb 4 2018, 06:30 PM
I've noted that due to an oversight on our part, both dns5.speedingbits.com and dns6.speedingbits.com were on the same node. They should not have been because they are the primary and secondary nameserver pair for many services.

Our priority right now is restoring dns5.speedingbits.com, which we should have complete in the next hour or two. This will restore service for many otherwise unrelated services that are currently showing as "down" for clients.

Posted by: andy Feb 4 2018, 07:31 PM
QUOTE (andy @ Feb 4 2018, 06:30 PM)
Our priority right now is restoring dns5.speedingbits.com, which we should have complete in the next hour or two. This will restore service for many otherwise unrelated services that are currently showing as "down" for clients.

This is still in progress.

We're creating a new system for dns5.speedingbits.com and will have the files restored within the next 60 minutes or so.

To facilitate this system being on a new high-availability platform, it needed to change IP addresses. We've changed the IP with the registrar already to begin the process of expiring the old information during the time while we're working to restore the files.

More information will be posted as it is known.

Thank you for your continued patience. We're working as quickly as possible to restore service for all those affected.

Posted by: andy Feb 4 2018, 08:30 PM
QUOTE (andy @ Feb 4 2018, 06:30 PM)
Our priority right now is restoring dns5.speedingbits.com, which we should have complete in the next hour or two. This will restore service for many otherwise unrelated services that are currently showing as "down" for clients.

The new dns5.speedingbits.com has modern versions of all system software (Apache, PHP, MySQL), so bear with us while we adapt the 10+ year old code to work on the new platform.

Our first priority will be to restore all authoritative zones (domains) serviced by this name server to get all of them functional again. Once we have done that, we'll begin work on making the web interface accessible and functional again.

We're currently working on transferring the files from our backup system to the new dns5.speedingbits.com.

Posted by: andy Feb 4 2018, 09:54 PM
Please stop texting and asking me to review tickets. They're likely all related to this issue at the moment. The few that aren't will be addressed by ticket support. Any tickets escalated to me will be delayed until further notice.

I won't be reviewing tickets or responding to texts until the resolution of all critical issues is complete. This will likely resolve your issue anyway.

At present, I'm the only one working this issue since nobody else has the technical qualifications to be of any use to the process. Unfortunately, I'm also off-hours support, so off-hours support is being ignored until the critical issues are resolved. The constant interruptions are substantially slowing the resolution, so I'll be handing off-hours support to a non-technical individual who will be responsible for telling people to check the forum for updates.

If you have an issue that you believe is not related to this set of issues, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php .

Please don't take random actions on your own in an attempt to "fix" this if you don't know what you're doing. You'll likely make a bigger mess that will have to be resolved at a later time. You're far better off just sitting tight or opening a ticket as indicated above.

No further responses to individual inquiries by text, phone, or email will be answered until further notice. It may appear that nothing is happening, but, rest assured, I'm working as quickly as possible to resolve ****ALL**** issues in the manner best suited to help as many people as possible as soon as possible.

If it appears nothing is happening because you aren't seeing constant updates on this thread, set that thought aside. I'm working on it as quickly as possible and will not leave anyone hanging any longer than is absolutely necessary.

Posted by: andy Feb 4 2018, 10:37 PM
dns5.speedingbits.com files are restored and adapted to the new system. Currently reviewing logs to vet the configs before making it reachable by the public. It will take 10 to 15 minutes to review the logs and if no problem are found, we'll unfirewall it to allow access by the public.

Posted by: andy Feb 4 2018, 10:51 PM
Aside from a few minor issues that can be resolved later (doesn't impact any active clients), the logs for dns5.speedingbits.com looked good and the configuration is working as expected.

We're now in the process of unfirewalling the new dns5.speedingbits.com.

Posted by: andy Feb 4 2018, 11:03 PM
dns5.speedingbits.com has been unfirewalled for several minutes now and it appears to be processing DNS queries normally. This should resolve a huge chunk of issues related to this outage of fs2.schmolie.com.

Access to sites using dns5.speedingbits.com and dns6.speedingbits.com may still be slower than usual since dns6.speedingbits.com isn't restored yet, but that is coming soon.

We're going to examine the remaining issues, triage them, and begin working them in order of priority.

Posted by: andy Feb 4 2018, 11:45 PM
The web interface for dns5.speedingbits.com has not yet been restored. We're saving that task for later since adding/changing zones is less critical than serving the existing zones.

The next thing in our list of priorities is to create a couple new DNS resolvers to replace the two that went down. Most systems and VPS on our network reference three DNS resolvers and two of them are currently down, resulting in slow access to anything that does DNS lookups such as SSH logins, ftp logins, and email logins. The services that rely on DNS resolution appear to be very slow to respond when only one of three DNS resolvers are up.

For some customer VPS that were already near to crashing for lack of CPU or RAM resources (you know who you are), this may have tipped them over the edge and caused their VPS to crash.

In total the DNS resolvers being down affects far more customers than just the ones on this node. They should be relatively quick to setup and put in service and have a large impact to our overall customer base.

Posted by: andy Feb 5 2018, 02:14 AM
The resolver dns4.schmolie.com has been replaced by a new system with a different IP.

The old ip was 66.29.143.240. The new IP is 66.29.136.240.

If you have a VPS or dedicated server and are technically inclined, you may edit your /etc/resolv.conf to reflect the new IP.

We'll be taking care of this for all the shared hosting systems and managed VPS immediately. For other systems, you can make the change on your own, open a ticket at https://cbp.speedingbits.com/billing/clientarea.php , or wait for us to make the change later.

We still have one more resolver to replace, which we'll be working on shortly.

Posted by: andy Feb 5 2018, 04:05 AM
We've finished updating the IP for dns4.schmolie.com in /etc/resolv.conf on every shared server system, internal systems, and managed VPS accounts.

We're now working on replacing dns5.schmolie.com.

Posted by: andy Feb 5 2018, 05:34 AM
The resolver dns5.schmolie.com has been replaced by a new system with a different IP.

The old ip was 66.29.144.140. The new IP is 66.29.136.140.

If you have a VPS or dedicated server and are technically inclined, you may edit your /etc/resolv.conf to reflect the new IP.

We'll be taking care of this for all the shared hosting systems and managed VPS immediately. For other systems, you can make the change on your own, open a ticket at https://cbp.speedingbits.com/billing/clientarea.php , or wait for us to make the change later.

Posted by: andy Feb 5 2018, 06:30 AM
We've finished updating the IP for dns5.schmolie.com in /etc/resolv.conf on every shared server system, internal systems, and managed VPS accounts.

We're now working on replacing dns6.speedingbits.com.

Posted by: andy Feb 5 2018, 01:15 PM
After a brief break for food and rest, we're continuing the configuration of the new dns6.speedingbits.com.

Posted by: andy Feb 5 2018, 02:41 PM
Currently working through some configuration problems preventing the primary and secondary (dns5.speedingbits.com and dns6.speedingbits.com) from synching properly.

We expect to have them synching properly in the next hour or so.

Posted by: andy Feb 5 2018, 03:20 PM
We've resolved the issue preventing dns6.speedingbits.com from synching with dns5.speedingbits.com.

We've unfirewalled dns6.speedingbits.com and it is now accessible to the public.

With the exception of the web interface on dns5.speedingbits.com for adding/deleting/modifying zones, all existing zones are being served properly by the new dns5.speedingbits.com and dns6.speedingbits.com at this time. The nameserver software, underlying OS, and hardware are now completely modern and upgraded from the prior systems which were nearing 12 years old.

We anticipate getting the web interface on dns5.speedingbits.com for adding/deleting/modifying zones functioning again either later today or tomorrow. There are still several other higher priority issues to be addressed first.

Posted by: andy Feb 5 2018, 03:34 PM
Next on the priority list is replacement of the system that handles outbound email from ohp0.speedingbits.com.

We expect that replacing this system will take approximately two hours and will restore outbound email service (web forms, notifications from scripts, etc.) for the customers with hosting accounts on ohp0.speedingbits.com.

Posted by: andy Feb 5 2018, 05:24 PM
If you're having trouble with email delivery on your VPS, as indicated previously in this thread, please edit your /etc/resolv.conf by logging in by SSH as root and editing the file with vi, vim, pico, nano, or your other favorite unix/linux text editor (don't copy the file to another system and edit because it may make your VPS unbootable or otherwise malfunction).

Change 66.29.143.240 to 66.29.136.240.
Change 66.29.144.140 to 66.29.136.140.

Save the changes and exit the text editor.

If you do not see an immediate result, restart your MTA service (sendmail, exim, postfix, etc.). If that does not resolve it, please reboot your VPS to ensure any old cached information about the old resolvers is gone.

Also, as indicated previously, if you're uncomfortable doing these tasks on your VPS, please open a ticket to get assistance (can be done at https://cbp.speedingbits.com/billing/clientarea.php ) . Be specific. General tickets like "Help!" and "Mail not working" will take longer for the ticket support team to assist. Reference this post if you have to.

Posted by: andy Feb 5 2018, 05:53 PM
QUOTE (andy @ Feb 5 2018, 03:34 PM)
Next on the priority list is replacement of the system that handles outbound email from ohp0.speedingbits.com.

We expect that replacing this system will take approximately two hours and will restore outbound email service (web forms, notifications from scripts, etc.) for the customers with hosting accounts on ohp0.speedingbits.com.

This system is now integrated and functioning. We're going to continue monitoring the logs for a bit to ensure all is well. There are messages in the queue on ohp0.speedingbits.com which will be retried during the next scheduled queue run.

So far, we've seen several legitimate messages and a lot of spam pass through for forwarded email accounts on ohp0.speedingbits.com, which isn't surprising since the ratio of legitimate email to spam is very low for many email addresses.

At this time, we believe outbound email (or forwarded email) on ohp0.speedingbits.com should be operating normally and email that was delayed should go through soon.

Posted by: andy Feb 5 2018, 06:32 PM
A few words about the shared hosting accounts (3 of them) and the 16 customer VPS that are not yet restored and our plan for handling those.

First, a brief explanation why we cannot simply restore your hosting account or VPS exactly as it was.

We had two virtuozzo nodes that could handle 64-bit OS's. We consolidated both of them to one and reinstalled XenServer on the node we emptied as the beginning of a migration process that would have carefully, calmly, and orderly migrated all of the old hosting and VPS accounts from those two nodes to a new modern virtualization solution and a modern OS. We understood that there was a small chance that the lack of redundant nodes could result in a situation in which a node failure of the remaining system could be problematic. Well, that roll of the dice, in spite of the low chance of problems, didn't work out in our (or your) favor, so we're forced to do the transition in a hurried and less orderly fashion.

Due to SWSoft (the company that made virtuozzo) changing hands 4 times (that we know of before we lost track of their current company name), we cannot relicense virtuozzo on a different machine or repair and relicense the same machine, thus making it impossible to create a system which we could restore. This is the unfortunately reality when the company that supports your commercial software flakes out and renigs on their obligations.

We recognize that this is a huge problem for some and an inconvenience for others, so here's what we're doing to move this process forward and get everyone running again.

We're crediting every MDH account or VPS account that was on fs2.schmolie.com for a full month's service.

If you had an MDH account (shared hosting), you'll be getting a brand new WebPro2 account and a WebPro2 Additional Domain account to equal the number of domains you were hosting on mdh0.speedingbits.com. We'll restore the site files to the WebPro2 account and recreate any email accounts. If we have control over the authoritative DNS servers listed for your domain, we'll then change DNS to point to the new server.

If you had a mid-range to high-end virtuozzo based VPS, it will be replaced with our absolute latest and best technology. You'll get a VM with automated high-availability failover between multiple nodes, blazing fast storage, CentOS 7, and CPanel/WHM. It will be on the same platform as our yet unreleased next gen VPS plans, the hwVCI. The platform has been well tested over the last few years and we trust it to run our own web site, billing/ticket systems, numerous DNS servers, and the shared server for WebPro2 accounts, to name a few. Your VM will have better specifications than your old VPS and will be priced the same as your old VPS. We'll restore any files we have to the new VPS account and recreate any email accounts. If we have control over the authoritative DNS servers listed for your domains, we'll then change DNS to point to the new server.

If you had a budget or economy oriented VPS, it will be replaced with an e-hwVPS running CentOS 6, our current budget oriented VPS offering. Discounts will be applied to the base price of the service to match the same price as your prior service. If you are in need of a control panel, we recommend Webmin/Virtualmin be installed, which is free and open source. If you'd like CPanel/WHM, it is available for $15/month and $15 setup, but only on freshly provisioned e-hwVPS. CPanel cannot be added later if you've installed any server software or configured away from defaults. We'll restore any files we have to the new e-hwVPS account and recreate any email accounts. If we have control over the authoritative DNS servers listed for your domains, we'll then change DNS to point to the new server.

We're going to be working through creating the various accounts and restoring files continuously until complete. Please bear with us as we complete this forced transition.

Posted by: andy Feb 5 2018, 08:08 PM
Some have had questions, so I'll answer them here for everyone since they're asked by multiple people:

1. Will my IP stay the same? No. You'll get a new IP, hence the need for DNS changes mentioned above. None of our modern systems are attached to our network in the same fashion as we did on this 12 year old system. Long story short, no, it is not possible to keep the same IP.

2. Is my account one of the shared hosting accounts or VPS mentioned? If your shared hosting account was on mdh0.speedingbits.com, yes (only three clients still on that system). If you normally login to HSPC ( https://manage.speedingbits.com/cp ) to manage your VPS and your VPS shows "offline" (be sure you're looking at VPS and not an anti-spam subscription in the "systems" tab), then yes - otherwise, no.

3. How will you notify me since my email address isn't working? We'll open a ticket on your account when the new system is ready for your use. You can login at https://cbp.speedingbits.com/billing/clientarea.php to see open tickets. We'll also post to this forum thread when we've finished with each type of account, which you can use as an indication you can go look for your ticket with your info.

4. Can I get configs restored from my old VPS to the new VPS? Yes and no. Yes, we'll put the files on a conspicuous location on your new VPS. However, there may be enough difference between your old VPS and your new VPS that the configs will not work directly and may require significant adaptation to make them work on the new VPS.

Posted by: andy Feb 6 2018, 10:36 AM
If you had an "Economy VPS COS4x64 2008" plan type for your VPS and are affected (your VPS shown as "offline" in HSPC), if you haven't already, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php , request your replacement e-hwVPS, and let us know if you want CPanel/WHM.

It does cost extra ($15 setup, $15/month), but if you want to have it, it needs to be decided up front before any configurations are made to the freshly provisioned e-hwVPS.

Your other options for controlling the e-hwVPS are command line via SSH (free, difficult for inexperienced), Virtualmin (free and paid versions, relatively easy, https://www.virtualmin.com/download.html ), or any other control panel of your choosing that will install on CentOS 6 and run with 1GB of RAM.

Many control panels, including Virtualmin, require that they are installed on a freshly installed empty system, so keep that in mind. Your control panel options narrow sharply once you start manually installing and configuring software on your own.

If we don't hear from you within the next few hours, we'll provision an e-hwVPS without CPanel and upload the backups into the e-hwVPS under /home.

Even then, we can still add CPanel/WHM as long as you haven't installed or configured any software yet.

Posted by: andy Feb 6 2018, 11:57 AM
We're working through restores as quickly as possible, but there are hundreds of GB and millions of files to restore. Right now, the bottleneck is the speed of our off-site backup system while doing restores. Suffice it to say, it's very busy at the moment.

It is impossible to give an ETA on individual restores since they are done sequentially and at different times, but it's looking like we'll be finished by this evening.

Posted by: andy Feb 6 2018, 07:57 PM
Restores are still proceeding. The bottleneck continues to be the performance of the server that handles backups. We're investigating possible methods to speed up the process.

Posted by: andy Feb 6 2018, 11:24 PM
We fixed the issue that was causing the backup server to be the bottleneck, which moved the bottleneck to the customers' VPS receiving the restore file.

After much experimenting with ways to speed up the transfer process, it was decided to upload the restore files to an interim staging server that's local to the nodes containing the customers' VPS rather than trying to go straight from the off-site backup system to the customers' VPS.

This resulted in a 8x speedup in transfer rate which will cut substantial time off the process.

Restores are still in progress.

Posted by: andy Feb 7 2018, 02:13 AM
The speedups allowed us to complete several very large restores tonight.

The remaining restores are a combination of small restores and complex restores that will take some time to work through. We have a prioritized list and are working through it.

Still on our to-do list is to restore the web interface for dns5.speedingbits.com. We realize that customers have not had access to change/add/delete zones on that system for several days. If you have an urgent DNS entry that needs to be made or changed, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can do it for you manually. If you are able to wait until the web interface is up, please make your entries at that time. We only have very limited time and have to reserve it for urgent issues until everyone's sites are up again.

As we make more progress, we'll post it here.

Posted by: andy Feb 7 2018, 05:43 PM
We're in the process of sorting out the MDH accounts and gathering the web site files from backups.

After we create replacement accounts, upload the files from backups, and recreate all email addresses on the new account, we'll change DNS for each to point to the new server and get in touch with each client to ensure they know how to access the new account.

Posted by: andy Feb 8 2018, 01:25 AM
We've finished restoring the MDH accounts to their new WebPro2 accounts.

We still have several VPS accounts left to restore, most of which we're waiting to hear back from the customer about their choice of control panel before we create the new VPS for them and begin the file restore process.

If you haven't been restored yet and you're not sure why, please login to the billing control panel at https://cbp.speedingbits.com/billing/clientarea.php , review any tickets we've opened for you on the topic and respond with the information requested so we can proceed.

If you've already responded, thank you. You're in the queue to be restored and we'll do so as soon as we can.

Posted by: andy Feb 9 2018, 01:49 AM
We're currently working through some of the more complex restores.

Some of the issues we're working through for certain clients:
  • digging up old mysql versions to restore databases from binary files because regular database dumps were not setup by the customer for their databases
  • locating 9 year old documentation (from employees who no longer work here) on mysql root passwords or going through mysql root password reset procedures
  • working out syntax differences between a mysql dump from a 9 year old version of mysql and a modern version of mysql
  • working out compatibility issues with old PHP code because there's a 9 year difference in versions between what they were using and what is current

That's just a small sampling of the tasks we're working through and a big part of why it is taking so long.

For those that never responded to their ticket with a choice of control panel for their e-hwVPS, most of those were recreated today and we'll hopefully be able to get those files uploaded for those clients tonight or tomorrow. The VPS that were recreated today had little to no traffic over the last month, so we question whether they were even in use any more. It was for this reason that they were delayed until after more active clients had been restored.

The process continues...

Posted by: andy Feb 9 2018, 05:19 PM
We're in the process of provisioning the four VPS to replace the four "High Performance VPS COS4x64 2008" services that were affected. These are also getting CPanel/WHM installed before files are restored.

This work is ongoing and will continue through the weekend until complete.

Posted by: andy Feb 10 2018, 12:10 AM
Holy lack of sleep Batman!

We're mostly done with this restore. There are still a few VPS accounts that either aren't being used or are otherwise not noticed by the clients that have not been restored. Prior to the outage, however, none of these had any significant traffic, so it doesn't look like they were being used.

We'll be working to get in touch with those clients.

If you're still having trouble related to this outage (or any kind of trouble, really), please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php .

We are aware that the web interface to dns5.speedingbits.com is still down and that is next on our list to restore. We'll begin that Sunday or Monday (Feb. 11 or 12).

We will continue posting updates on this thread as needed, but they will be less frequent going forward since there are very few still affected and we're probably just talking to the wind now.

Posted by: andy Feb 13 2018, 01:21 AM
We made some progress on the web interface for dns5.speedingbits.com.

We adapted the automation that keeps the zone names synched between the primary and secondary and got that running.

We've also located the backup files and database for the web interface and created a restore plan.

We still need to restore the database to an old version of MySQL, export the database, import/adapt the database for the new version of MySQL/MariaDB, restore the web interface files, adapt the web interface configuration to the new system, test the various functions, debug any PHP version compatibility issues, generate a new valid SSL certificate, and open the firewall so customers can begin using the system again.

Barring unexpected complications, we should have the dns5.speedingbits.com web interface back in service by Wednesday evening.

Posted by: andy Feb 15 2018, 12:54 PM
We are continuing to make progress on restoring the web interface for dns5.speedingbits.com.

After much wrangling, we finally were able to get a working MySQL installation the same version as the original dns5.speedingbits.com was using.

We're now working through restoring our database files to that system from backups so we can take a MySQL dump that can be imported into the newer MySQL version on the new dns5.speedingbits.com.

As indicated in my Feb. 13 post on this thread, there are several additional steps after this that will still need to be done before the web interface is completely restored.

Posted by: andy Feb 15 2018, 03:35 PM
We've successfully restored the mysql database for the dns5.speedingbits.com web interface and are now working on adapting the php code and perl scripts for the web interface so that it will function on this much newer environment.

As indicated in my Feb. 13 post on this thread, there are several additional steps after this that will still need to be done before the web interface is completely restored.

Posted by: andy Feb 16 2018, 05:27 PM
We're still working on the web interface for dns5.speedingbits.com.

Again, while the web interface is down, if you have a need for a new DNS entry or a change to an existing DNS entry on dns5.speedingbits.com, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can make the entry or change manually on your behalf.

Please don't just stew until you scream. Ask for help. It's readily available and willingly offered.

Posted by: andy Mar 2 2018, 06:14 PM
After numerous delays and interruptions from other higher priority projects, we've finally made some progress with the web interface of dns5.speedingbits.com.

We've converted all the configurations to work on the new system. We've setup a working PERL environment and all related modules for the backend automation. We've gotten some of the backend automation working.

Still to do:
  • Get all the PHP code working in PHP 7
  • Finish getting the remainder of the automation related code reworked for the new system
  • SSL certificate provisioning
  • Final testing
  • Setup daily off-site backups
  • Adjust 24/7 monitoring systems

It's still too early to give an ETA for the web interface to be functional again. Getting code working in PHP 7 that was originally written for an older version of PHP can be quick or it could take a great amount of time, depending entirely on the code and how it was written. We won't know how long it will take until we're done, but we're expecting it could be in the range of one work day to several weeks, depending on numerous factors, many not in our control.

This code was written by a third party and the related code project was abandoned by the original author in 2005. The project was setup in such a way that others could not continue the project independently, so it became orphaned. It was at least open source so that we could continue to use and maintain the code we have in spite of no updates from the original author or the community (no project community was allowed to exist by the original author).

Again, while the web interface is down, if you have a need for a new DNS entry or a change to an existing DNS entry on dns5.speedingbits.com, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can make the entry or change manually on your behalf.

Posted by: andy Apr 3 2018, 04:22 PM
QUOTE (andy @ Mar 2 2018, 06:14 PM)
Still to do:
  • Get all the PHP code working in PHP 7
  • Finish getting the remainder of the automation related code reworked for the new system
  • SSL certificate provisioning
  • Final testing
  • Setup daily off-site backups
  • Adjust 24/7 monitoring systems
It's still too early to give an ETA for the web interface to be functional again. Getting code working in PHP 7 that was originally written for an older version of PHP can be quick or it could take a great amount of time, depending entirely on the code and how it was written. We won't know how long it will take until we're done, but we're expecting it could be in the range of one work day to several weeks, depending on numerous factors, many not in our control.

This code was written by a third party and the related code project was abandoned by the original author in 2005. The project was setup in such a way that others could not continue the project independently, so it became orphaned. It was at least open source so that we could continue to use and maintain the code we have in spite of no updates from the original author or the community (no project community was allowed to exist by the original author).

Again, while the web interface is down, if you have a need for a new DNS entry or a change to an existing DNS entry on dns5.speedingbits.com, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php with all the relevant details so that we can make the entry or change manually on your behalf.

Regarding the dns5.speedingbits.com web interface, we've completed the SSL configuration and certificate provisioning.

We're currently working through debugging the old PHP code on the new PHP version. So far, it looks like we have to rewrite all of the mysql_ related functions to support the change in the way PHP accesses MySQL between PHP 4 and PHP 7. There are also a few other error messages we're still tracking down.

With a few more hours of concentrated time and effort, I should be able to complete this major update to the web interface of dns5.speedingbits.com. While rare that I get several hours uninterrupted time, recent changes to technical support duties I previously handled should make this more likely more often.

Posted by: andy Apr 4 2018, 04:42 PM
We've made more progress on the web interface of dns5.speedingbits.com. The database related functions are working correctly now and we've replaced several function calls with their PHP 7 equivalent.

We're probably about 30% done with replacing function calls that no longer exist in PHP 7. We'll be continuing this work tomorrow.

Posted by: andy Apr 5 2018, 05:09 PM
The web interface for dns5.speedingbits.com is nearly complete.

We've finished porting the code to the latest version of PHP, but we're still working to modify the code of the web interface to properly support long DKIM keys in TXT records. This is important for anyone making use of Google email services with their domain since Google requires people to use extremely long DKIM keys that break older DNS systems.

We expect to have this modification complete tomorrow.

Posted by: andy Apr 6 2018, 12:12 AM
The web interface for dns5.speedingbits.com is complete.

Some improvements over the old version:
  • Running on fast hardware
  • Running on a modern OS with modern PHP, MySQL, and nameserver software
  • SSL implemented - no more insecure logins
  • Fixed several longstanding bugs
  • Now supports Google's gigantic DKIM keys

One important change in how this operates versus previously:
TXT records now require that you put a quotation mark (") at the beginning and end of the text you enter as the value for the TXT record. Previously, this was done for you, but to accommodate special formatting required to handle large DKIM keys, we had to change this.

To use a DKIM key larger than 255 bytes in length, you'll need to break it up into 253 character or less chunks, including spaces, and place quotation marks around each chunk. Insert a space between each quoted chunk of text. The last chunk of text needs a semicolon (;) just before the final quotation mark. For instance:
CODE

"v=DKIM1; k=rsa; p=a/big/long/chunk/of/text" "another/big/long/chunk/of/text;"


If you had an account on the old system, it is now active on this new system. Feel free to use it as you did before.

If you encounter any problems with the system, please open a ticket at: https://cbp.speedingbits.com/billing/clientarea.php

Powered by Invision Power Board (http://www.invisionboard.com)
© Invision Power Services (http://www.invisionpower.com)