Migrating huge WordPress sites reliably

None of the WordPress migration or backup and restore plugins can handle huge sites reliably – you’ll run into timeout issues etc. Here’s how to migrate your huge sites reliably. The guide below presumes you have an origin WordPress server and a destination WordPress-ready (PHP+MySQL+Nginx) server.

Use SSH and Screen

You need to be able to SSH onto your server to do this reliably. You also need to be able to rely on the process completing, even if you switch your computer off or your session gets terminated (like if your network connection drops). Screen is what we need for that purpose – it creates a detachable screen inside SSH that will keep running even when we disconnect from the server.

To use screen, just run:

screen

(then hit enter a few times)

You can then run commands then hit CTRL+A CTRL+D and it will disconnect from the ‘screen’.

You can reattach detached screens using:

screen -r

If there’s more than one detached screen, it’ll instead list the detached screens like this:

And you can reconnect by copying one of the session identifiers into a command like this:

screen -d -r 4674.pts-3.p1

Create a MySQL backup reliably

This technique always works, provided you have enough storage space. If you need more storage space, add a Digital Ocean volume temporarily. If your database is huge, run Screen first so the command doesn’t get interrupted if you disconnect.

mysqldump -uWPDBUSER -p WPDBNAME > '/root/dump.sql'

Transfer all your files across

SSH into your destination server, install rsync, then use rsync to run the transfer. The advantages of rsync over FTP are many – it doesn’t break with long path names, it retries if the connection drops, it continues until complete – it works. If you have a lot of files, you should run screen first, so rsync will just keep going even if your SSH session disconnects.

sudo apt-get install rsync
rsync -chavzP --stats [email protected]:/var/www/ /var/www/

(replace root@ with username@ if you’re not using the root user)

Also grab the dump.sql file:

rsync -chavzP --stats [email protected]:/root/dump.sql /root/dump.sql

If you need to exclude a particular folder, you can do so – but bear in mind the folder to exclude is relative to the path you entered, e.g. to exclude the /var/www/yourdomain.com/wp-content/uploads folder use:

rsync -chavzP --stats --exclude 'yourdomain.com/wp-content/uploads' [email protected]:/var/www/ /var/www/

Note: If the origin server uses SSH keys, then generate a key on your destination server and insert the /root/.ssh/id_rsa.pub key into /root/.ssh/authorized_keys on the source server so you can authenticate.

If you can’t SSH to the old site (shared hosting?) there is probably cPanel available, so get a cPanel backup created from the originating site and FTP that over instead.

Restoring your database

You can use a plugin like UpdraftPlus if your databases are small enough. But if you’re up at a few GB or near TB or past then you should be using mysql from the command line to avoid timeouts and ensure completion.

If you haven’t already, create a database user for the database:

(from inside mysql)

create database dbname;
grant all privileges on dbname.* to "username"@"localhost" identified by "dbpassword";

Then back in SSH/bash, presuming you have a dump.sql file, we need to replace top line of the dump file with the database you are restoring to.

sed -i -e '1iUSE dbname;\' /root/dump.sql
mysql < /root/dump.sql

Configure Nginx to look at the new files

Finally, modify your NginX config to point to the new files.

vi /etc/nginx/sites-available/yourdomain.conf

(or whatever it’s called)

change the folder in your config file to point to the new folder that’s present after rsync is complete.

Note: You may have different nginx users for source and destination web servers – to alter the owner of the web files, you can use something like this:

chown wordpress:wordpress /var/www/yourfolder/ -R

Changing Domain Names on huge sites

If you’re also changing domain names, e.g. if you’re creating a staging or dev site, or maybe temporarily moving your site to different hosting to check performance then you can’t use WordPress search & replace plugins and even the interconnectit web-based script will break on huge sites.

So, use the command line version of interconnectit. Download using the following:

cd /root/
git clone https://github.com/interconnectit/Search-Replace-DB.git
cd Search-Replace-DB/

Now you can run search and replace in your database – change the text in CAPS below with your own specific details:

 php srdb.cli.php -h localhost -n DBNAME -u DBUSER -p 'PASSWORD' -s 'ORIGINALDOMAIN' -r 'NEWDOMAIN'

Note: The above command includes placing the database password in the command line. That means there is the danger of it ending up in the history and you don’t want this. To avoid the command going in the history list, ensure you put a single space before the command – i.e. a space before the letters php. This signals to linux to not log this command.

Migrating SSL certificates

You can either copy them across manually, if you’re keeping the same domain name, or create new ones using Letsencrypt.

git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
/opt/letsencrypt/letsencrypt-auto certonly -a webroot --webroot-path=/var/www/acme/ -d peepgo.com -d www.peepgo.com --register-unsafely-without-email --agree-tos

The new SSL key files will be in /etc/letsencrypt/live/yourdomain/fullchain.pem and /etc/letsencrypt/live/yourdomain/privkey.pem.

To add them, modify your /etc/nginx/sites-available/yourdomain.conf file (or varnish.yourdomain… file) and find the current two lines which specify the keys for the old domain/server:

 ssl_certificate /etc/letsencrypt/live/yourdomain/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/yourdomain/privkey.pem;

Modify the paths for those 2 lines in the nginx sites-available file to match the location of your newly generated letsencrypt SSL keys. Restart nginx to get it to pick up the new key files.

service nginx restart

Fixing WordPress file and folder permissions

Some problems can occur. e.g. if the originating host messed around with wp-config.php then they might have altered ABS definitions. Or, permissions might have been set badly on the originating host.

Useful commands below to fix WordPress folder permissions:

chown wordpress:wordpress /var/www/domain/ -R

find /var/www/domain/ -type d -exec chmod 755 {} \;

find . -type f -exec chmod 644 {} \;

Debugging & Troubleshooting any issues

If you are having issues, there are a number of log files you can examine:

tail /var/log/nginx/varnish.yourdomain.conf.error.conf
tail /var/log/nginx/yourdomain.conf.error.conf
tail /var/log/nginx/error.conf
tail /var/log/php7.0-fpm.log

Also, you can edit wp-config.php and add these lines:

define('WP_DEBUG', true);
define('WP_DEBUG_LOG', true);
define('WP_DEBUG_DISPLAY', false);

Then try and load your broken home page. Once it has failed, you can look in the new wp-content/debug.log file:

tail /var/www/yourdomain.com/wp-content/debug.log

Summary

The above shows the guaranteed method to migrate WordPress sites of any size. If you have any questions, or think I’ve missed anything, let me know below.

Chat to me

Dave Hilditch

Founder at WP Intense
Dave has been programming since 6 years old and has been developing WordPress plugins, themes and websites since 2010. In the past he built the browse view technology for Skyscanner and now he helps clients with interesting website challenges.

He is always on at least one of his computers when he's awake, so get in touch and he'll get right back to you.
Chat to me