Migrating huge WordPress sites reliably

• Author
• Recent Posts

Talk to me

Dave Hilditch

Founder at WP Intense

Author of the WPI performance guides and creator and coder of the plugins sold by WP Intense.

Talk to me

Latest posts by Dave Hilditch (see all)

• Running huge imports with WP All Import reliably (yes, even if you use Cloudflare!) - September 19, 2023
• New SQL enhancement for Scalability Pro to fix WooCommerce long-running query in the product-hero block - August 31, 2023
• Create a static favicon.ico to avoid surplus PHP requests for rush traffic - July 25, 2023

You can chat to all our authors on our Discord

None of the WordPress migration or backup and restore plugins can handle huge sites reliably – you’ll run into timeout issues etc. Here’s how to migrate your huge sites reliably. The guide below presumes you have an origin WordPress server and a destination WordPress-ready (PHP+MySQL+Nginx) server.

Use SSH and Screen

You need to be able to SSH onto your server to do this reliably. You also need to be able to rely on the process completing, even if you switch your computer off or your session gets terminated (like if your network connection drops). Screen is what we need for that purpose – it creates a detachable screen inside SSH that will keep running even when we disconnect from the server.

To use screen, just run:

screen

(then hit enter a few times)

You can then run commands then hit CTRL+A CTRL+D and it will disconnect from the ‘screen’.

You can reattach detached screens using:

screen -r

If there’s more than one detached screen, it’ll instead list the detached screens like this:

And you can reconnect by copying one of the session identifiers into a command like this:

screen -d -r 4674.pts-3.p1

Create a MySQL backup reliably

This technique always works, provided you have enough storage space. If you need more storage space, add a Digital Ocean volume temporarily. If your database is huge, run Screen first so the command doesn’t get interrupted if you disconnect.

mysqldump -uWPDBUSER -p WPDBNAME > '/root/dump.sql'

Transfer all your files across

SSH into your destination server, install rsync, then use rsync to run the transfer. The advantages of rsync over FTP are many – it doesn’t break with long path names, it retries if the connection drops, it continues until complete – it works. If you have a lot of files, you should run screen first, so rsync will just keep going even if your SSH session disconnects.

sudo apt-get install rsync
rsync -chavzP --stats  root@sourceip:/var/www/ /var/www/

(replace root@ with username@ if you’re not using the root user)

Also grab the dump.sql file:

rsync -chavzP --stats  root@sourceip:/root/dump.sql /root/dump.sql

If you need to exclude a particular folder, you can do so – but bear in mind the folder to exclude is relative to the path you entered, e.g. to exclude the /var/www/yourdomain.com/wp-content/uploads folder use:

rsync -chavzP --stats --exclude 'yourdomain.com/wp-content/uploads' root@sourceip:/var/www/ /var/www/

Note: If the origin server uses SSH keys, then generate a key on your destination server and insert the /root/.ssh/id_rsa.pub key into /root/.ssh/authorized_keys on the source server so you can authenticate.

ssh-keygen
cat /root/.ssh/id_rsa.pub

If you can’t SSH to the old site (shared hosting?) there is probably cPanel available, so get a cPanel backup created from the originating site and FTP that over instead.

Restoring your database

You can use a plugin like UpdraftPlus if your databases are small enough. But if you’re up at a few GB or near TB or past then you should be using mysql from the command line to avoid timeouts and ensure completion.

If you haven’t already, create a database user for the database:

(from inside mysql)

create database dbname;
CREATE USER 'username'@'localhost' IDENTIFIED WITH mysql_native_password BY 'CHOOSEASTRONGPASSWORD';
GRANT ALL PRIVILEGES ON dbname.* TO 'username'@'localhost';

Note: If you are migrating to a cluster, run this shell command before you attempt the restore, or if the restore fails. Clusters cannot use MyISAM.

sed -i 's/MyISAM/InnoDB/g' dump.sql
sed -i 's/^UNLOCK TABLES/-- UNLOCK TABLES/g' mysql.sql
sed -i 's/^LOCK TABLES/-- LOCK TABLES/g' mysql.sql

Then run the import.

mysql dbname < /root/dump.sql

Configure Nginx to look at the new files

Finally, modify your NginX config to point to the new files.

vi /etc/nginx/sites-available/yourdomain.conf

(or whatever it’s called)

change the folder in your config file to point to the new folder that’s present after rsync is complete.

Note: You may have different nginx users for source and destination web servers – to alter the owner of the web files, you can use something like this:

chown wordpress:wordpress /var/www/yourfolder/ -R

Changing Domain Names on huge sites

If you’re also changing domain names, e.g. if you’re creating a staging or dev site, or maybe temporarily moving your site to different hosting to check performance then you can’t use WordPress search & replace plugins and even the interconnectit web-based script will break on huge sites.

So, use the command line version of interconnectit. Download using the following:

cd /root/
git clone https://github.com/interconnectit/Search-Replace-DB.git
cd Search-Replace-DB/

Note: If you are using PHP 7, you’ll need to grab the latest version from here and upload it to your server:

https://interconnectit.com/products/search-and-replace-for-wordpress-databases/

Now you can run search and replace in your database – change the text in CAPS below with your own specific details:

 php srdb.cli.php -h localhost -n DBNAME -u DBUSER -p 'PASSWORD' -s 'ORIGINALDOMAIN' -r 'NEWDOMAIN'

Note: The above command includes placing the database password in the command line. That means there is the danger of it ending up in the history and you don’t want this. To avoid the command going in the history list, ensure you put a single space before the command – i.e. a space before the letters php. This signals to linux to not log this command.

Migrating SSL certificates

You can either copy them across manually, if you’re keeping the same domain name, or create new ones using Letsencrypt.

git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
/opt/letsencrypt/letsencrypt-auto certonly -a webroot --webroot-path=/var/www/acme/ -d peepgo.com -d www.peepgo.com --register-unsafely-without
email --agree-tos

The new SSL key files will be in /etc/letsencrypt/live/yourdomain/fullchain.pem and /etc/letsencrypt/live/yourdomain/privkey.pem.

To add them, modify your /etc/nginx/sites-available/yourdomain.conf file (or varnish.yourdomain… file) and find the current two lines which specify the keys for the old domain/server:

 ssl_certificate /etc/letsencrypt/live/yourdomain/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/yourdomain/privkey.pem;

Modify the paths for those 2 lines in the nginx sites-available file to match the location of your newly generated letsencrypt SSL keys. Restart nginx to get it to pick up the new key files.

service nginx restart

Additional SSL migration steps when migrating from proxy-SSL

If you are migrating from WP-Engine, or maybe you’re using Cloudflare, your site may actually be HTTP instead of HTTPS with the SSL only existing between your customers and the proxy.

If that is the case, you can end up with redirect errors. To avoid this, you can do the following:

1. Install the Really Simple SSL plugin on your new server and activate it
2. Test by altering your own hosts file to point the domain at the new server
3. You will need to re-activate the plugin after you have re-copied the DB, e.g. if you’re allowing a couple of minutes downtime between SQL backup and restore
4. Once the migration is complete and you have switched domains, you can permanently fix the SSL issues by running the following command:

 php srdb.cli.php -h localhost -n DBNAME -u DBUSER -p 'PASSWORD' -s 'http://originaldomain' -r 'https://newdomain'

You will need to run the above command from the folder you downloaded this program to earlier – probably in /root/Search-Replace-DB/

Once the search/replace is complete, you can deactivate the Really Simple SSL plugin and delete it.

Fixing WordPress file and folder permissions

Some problems can occur. e.g. if the originating host messed around with wp-config.php then they might have altered ABS definitions. Or, permissions might have been set badly on the originating host.

Useful commands below to fix WordPress folder permissions:

chown wordpress:wordpress /var/www/domain/ -R

find /var/www/domain/ -type d -exec chmod 755 {} \;

find . -type f -exec chmod 644 {} \;

Debugging & Troubleshooting any issues

If you are having issues, there are a number of log files you can examine:

tail /var/log/nginx/varnish.yourdomain.conf.error.conf
tail /var/log/nginx/yourdomain.conf.error.conf
tail /var/log/nginx/error.conf
tail /var/log/php7.0-fpm.log

Also, you can edit wp-config.php and add these lines:

define('WP_DEBUG', true);
define('WP_DEBUG_LOG', true);
define('WP_DEBUG_DISPLAY', false);

Then try and load your broken home page. Once it has failed, you can look in the new wp-content/debug.log file:

tail /var/www/yourdomain.com/wp-content/debug.log

Summary

The above shows the guaranteed method to migrate WordPress sites of any size. If you have any questions, or think I’ve missed anything, let me know below.