Powered by Invision Power Board


Pages: (3) 1 [2] 3   ( Go to first unread post ) Reply to this topicStart new topicStart Poll

> fs2.schmolie.com down (failed storage backplane), restoration completed
andy
Posted: Feb 5 2018, 04:05 AM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



We've finished updating the IP for dns4.schmolie.com in /etc/resolv.conf on every shared server system, internal systems, and managed VPS accounts.

We're now working on replacing dns5.schmolie.com.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 05:34 AM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



The resolver dns5.schmolie.com has been replaced by a new system with a different IP.

The old ip was 66.29.144.140. The new IP is 66.29.136.140.

If you have a VPS or dedicated server and are technically inclined, you may edit your /etc/resolv.conf to reflect the new IP.

We'll be taking care of this for all the shared hosting systems and managed VPS immediately. For other systems, you can make the change on your own, open a ticket at https://cbp.speedingbits.com/billing/clientarea.php , or wait for us to make the change later.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 06:30 AM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



We've finished updating the IP for dns5.schmolie.com in /etc/resolv.conf on every shared server system, internal systems, and managed VPS accounts.

We're now working on replacing dns6.speedingbits.com.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 01:15 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



After a brief break for food and rest, we're continuing the configuration of the new dns6.speedingbits.com.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 02:41 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



Currently working through some configuration problems preventing the primary and secondary (dns5.speedingbits.com and dns6.speedingbits.com) from synching properly.

We expect to have them synching properly in the next hour or so.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 03:20 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



We've resolved the issue preventing dns6.speedingbits.com from synching with dns5.speedingbits.com.

We've unfirewalled dns6.speedingbits.com and it is now accessible to the public.

With the exception of the web interface on dns5.speedingbits.com for adding/deleting/modifying zones, all existing zones are being served properly by the new dns5.speedingbits.com and dns6.speedingbits.com at this time. The nameserver software, underlying OS, and hardware are now completely modern and upgraded from the prior systems which were nearing 12 years old.

We anticipate getting the web interface on dns5.speedingbits.com for adding/deleting/modifying zones functioning again either later today or tomorrow. There are still several other higher priority issues to be addressed first.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 03:34 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



Next on the priority list is replacement of the system that handles outbound email from ohp0.speedingbits.com.

We expect that replacing this system will take approximately two hours and will restore outbound email service (web forms, notifications from scripts, etc.) for the customers with hosting accounts on ohp0.speedingbits.com.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 05:24 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



If you're having trouble with email delivery on your VPS, as indicated previously in this thread, please edit your /etc/resolv.conf by logging in by SSH as root and editing the file with vi, vim, pico, nano, or your other favorite unix/linux text editor (don't copy the file to another system and edit because it may make your VPS unbootable or otherwise malfunction).

Change 66.29.143.240 to 66.29.136.240.
Change 66.29.144.140 to 66.29.136.140.

Save the changes and exit the text editor.

If you do not see an immediate result, restart your MTA service (sendmail, exim, postfix, etc.). If that does not resolve it, please reboot your VPS to ensure any old cached information about the old resolvers is gone.

Also, as indicated previously, if you're uncomfortable doing these tasks on your VPS, please open a ticket to get assistance (can be done at https://cbp.speedingbits.com/billing/clientarea.php ) . Be specific. General tickets like "Help!" and "Mail not working" will take longer for the ticket support team to assist. Reference this post if you have to.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 05:53 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



QUOTE (andy @ Feb 5 2018, 03:34 PM)
Next on the priority list is replacement of the system that handles outbound email from ohp0.speedingbits.com.

We expect that replacing this system will take approximately two hours and will restore outbound email service (web forms, notifications from scripts, etc.) for the customers with hosting accounts on ohp0.speedingbits.com.

This system is now integrated and functioning. We're going to continue monitoring the logs for a bit to ensure all is well. There are messages in the queue on ohp0.speedingbits.com which will be retried during the next scheduled queue run.

So far, we've seen several legitimate messages and a lot of spam pass through for forwarded email accounts on ohp0.speedingbits.com, which isn't surprising since the ratio of legitimate email to spam is very low for many email addresses.

At this time, we believe outbound email (or forwarded email) on ohp0.speedingbits.com should be operating normally and email that was delayed should go through soon.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 06:32 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



A few words about the shared hosting accounts (3 of them) and the 16 customer VPS that are not yet restored and our plan for handling those.

First, a brief explanation why we cannot simply restore your hosting account or VPS exactly as it was.

We had two virtuozzo nodes that could handle 64-bit OS's. We consolidated both of them to one and reinstalled XenServer on the node we emptied as the beginning of a migration process that would have carefully, calmly, and orderly migrated all of the old hosting and VPS accounts from those two nodes to a new modern virtualization solution and a modern OS. We understood that there was a small chance that the lack of redundant nodes could result in a situation in which a node failure of the remaining system could be problematic. Well, that roll of the dice, in spite of the low chance of problems, didn't work out in our (or your) favor, so we're forced to do the transition in a hurried and less orderly fashion.

Due to SWSoft (the company that made virtuozzo) changing hands 4 times (that we know of before we lost track of their current company name), we cannot relicense virtuozzo on a different machine or repair and relicense the same machine, thus making it impossible to create a system which we could restore. This is the unfortunately reality when the company that supports your commercial software flakes out and renigs on their obligations.

We recognize that this is a huge problem for some and an inconvenience for others, so here's what we're doing to move this process forward and get everyone running again.

We're crediting every MDH account or VPS account that was on fs2.schmolie.com for a full month's service.

If you had an MDH account (shared hosting), you'll be getting a brand new WebPro2 account and a WebPro2 Additional Domain account to equal the number of domains you were hosting on mdh0.speedingbits.com. We'll restore the site files to the WebPro2 account and recreate any email accounts. If we have control over the authoritative DNS servers listed for your domain, we'll then change DNS to point to the new server.

If you had a mid-range to high-end virtuozzo based VPS, it will be replaced with our absolute latest and best technology. You'll get a VM with automated high-availability failover between multiple nodes, blazing fast storage, CentOS 7, and CPanel/WHM. It will be on the same platform as our yet unreleased next gen VPS plans, the hwVCI. The platform has been well tested over the last few years and we trust it to run our own web site, billing/ticket systems, numerous DNS servers, and the shared server for WebPro2 accounts, to name a few. Your VM will have better specifications than your old VPS and will be priced the same as your old VPS. We'll restore any files we have to the new VPS account and recreate any email accounts. If we have control over the authoritative DNS servers listed for your domains, we'll then change DNS to point to the new server.

If you had a budget or economy oriented VPS, it will be replaced with an e-hwVPS running CentOS 6, our current budget oriented VPS offering. Discounts will be applied to the base price of the service to match the same price as your prior service. If you are in need of a control panel, we recommend Webmin/Virtualmin be installed, which is free and open source. If you'd like CPanel/WHM, it is available for $15/month and $15 setup, but only on freshly provisioned e-hwVPS. CPanel cannot be added later if you've installed any server software or configured away from defaults. We'll restore any files we have to the new e-hwVPS account and recreate any email accounts. If we have control over the authoritative DNS servers listed for your domains, we'll then change DNS to point to the new server.

We're going to be working through creating the various accounts and restoring files continuously until complete. Please bear with us as we complete this forced transition.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 5 2018, 08:08 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



Some have had questions, so I'll answer them here for everyone since they're asked by multiple people:

1. Will my IP stay the same? No. You'll get a new IP, hence the need for DNS changes mentioned above. None of our modern systems are attached to our network in the same fashion as we did on this 12 year old system. Long story short, no, it is not possible to keep the same IP.

2. Is my account one of the shared hosting accounts or VPS mentioned? If your shared hosting account was on mdh0.speedingbits.com, yes (only three clients still on that system). If you normally login to HSPC ( https://manage.speedingbits.com/cp ) to manage your VPS and your VPS shows "offline" (be sure you're looking at VPS and not an anti-spam subscription in the "systems" tab), then yes - otherwise, no.

3. How will you notify me since my email address isn't working? We'll open a ticket on your account when the new system is ready for your use. You can login at https://cbp.speedingbits.com/billing/clientarea.php to see open tickets. We'll also post to this forum thread when we've finished with each type of account, which you can use as an indication you can go look for your ticket with your info.

4. Can I get configs restored from my old VPS to the new VPS? Yes and no. Yes, we'll put the files on a conspicuous location on your new VPS. However, there may be enough difference between your old VPS and your new VPS that the configs will not work directly and may require significant adaptation to make them work on the new VPS.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 6 2018, 10:36 AM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



If you had an "Economy VPS COS4x64 2008" plan type for your VPS and are affected (your VPS shown as "offline" in HSPC), if you haven't already, please open a ticket at https://cbp.speedingbits.com/billing/clientarea.php , request your replacement e-hwVPS, and let us know if you want CPanel/WHM.

It does cost extra ($15 setup, $15/month), but if you want to have it, it needs to be decided up front before any configurations are made to the freshly provisioned e-hwVPS.

Your other options for controlling the e-hwVPS are command line via SSH (free, difficult for inexperienced), Virtualmin (free and paid versions, relatively easy, https://www.virtualmin.com/download.html ), or any other control panel of your choosing that will install on CentOS 6 and run with 1GB of RAM.

Many control panels, including Virtualmin, require that they are installed on a freshly installed empty system, so keep that in mind. Your control panel options narrow sharply once you start manually installing and configuring software on your own.

If we don't hear from you within the next few hours, we'll provision an e-hwVPS without CPanel and upload the backups into the e-hwVPS under /home.

Even then, we can still add CPanel/WHM as long as you haven't installed or configured any software yet.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 6 2018, 11:57 AM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



We're working through restores as quickly as possible, but there are hundreds of GB and millions of files to restore. Right now, the bottleneck is the speed of our off-site backup system while doing restores. Suffice it to say, it's very busy at the moment.

It is impossible to give an ETA on individual restores since they are done sequentially and at different times, but it's looking like we'll be finished by this evening.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 6 2018, 07:57 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



Restores are still proceeding. The bottleneck continues to be the performance of the server that handles backups. We're investigating possible methods to speed up the process.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
andy
Posted: Feb 6 2018, 11:24 PM
Quote Post





Group: Advantagecom Staff
Posts: 4,310
Member No.: 9
Joined: 12-July 02



We fixed the issue that was causing the backup server to be the bottleneck, which moved the bottleneck to the customers' VPS receiving the restore file.

After much experimenting with ways to speed up the transfer process, it was decided to upload the restore files to an interim staging server that's local to the nodes containing the customers' VPS rather than trying to go straight from the off-site backup system to the customers' VPS.

This resulted in a 8x speedup in transfer rate which will cut substantial time off the process.

Restores are still in progress.


--------------------
Sincerely,
Andrew Kinney
CTO, Advantagecom Networks

Please do not private message me. My regular management duties preclude responding to every customer that sends me a support issue. Instead, post on the forum or contact tech support.
PMUsers Website
Top
0 User(s) are reading this topic (0 Guests and 0 Anonymous Users)
0 Members:

Topic Options Pages: (3) 1 [2] 3  Reply to this topicStart new topicStart Poll