Transfer files quickly from Amazon S3

• Author
• Recent Posts

Talk to me

Dave Hilditch

Founder at WP Intense

Author of the WPI performance guides and creator and coder of the plugins sold by WP Intense.

Talk to me

Latest posts by Dave Hilditch (see all)

• Running huge imports with WP All Import reliably (yes, even if you use Cloudflare!) - September 19, 2023
• New SQL enhancement for Scalability Pro to fix WooCommerce long-running query in the product-hero block - August 31, 2023
• Create a static favicon.ico to avoid surplus PHP requests for rush traffic - July 25, 2023

You can chat to all our authors on our Discord

It may be that you’re running a migration for someone, or you’ve decided to switch to using Digital Ocean volumes. Whatever the reason, downloading files from S3 using rsync + fuse is very slow. Here’s how to transfer files quickly.

apt install python-pip
pip install awscli --upgrade --user
pip install --upgrade pip
apt install awscli

Now, that’s AWS installed, you need to configure it with your access key and secret key.

aws configure

It’ll ask you for your access key and secret key, so paste them into the window. Once done, you can download your bucket.

aws s3 sync s3://mybucket /root/s3dest/

That’s it – it’ll download the entire bucket now. Performance-wise, I have not benchmarked this, but it’s about 4 or 5 times faster than using fuse + rsync.