For the past 2 weeks, WP Intense has been under a sustained attack from a hacker.
Someone, somewhere, has purchased a massive database of emails + passwords and has been trying to hack into our website using these emails. The scale of this is fairly enormous – in the past week alone, over 2 million hack-attempts were made.
Why the normal security measures don’t help
Normally, fail2ban would help by allowing us to ban these IP addresses using various traffic filters but this botnet is using hundreds of thousands of IP addresses, so there is very little benefit to banning IP addresses as the botnet just moves onto another IP address.
The amount of traffic was swamping our servers and hurting our regular user traffic, so something really needed to be done, either at the nginx or the fail2ban level.
Using something like Sucuri or WordFence wouldn’t help here either, since they involve a full load of PHP and WordPress which for this level of traffic is very heavy. I did try them, but neither helped and actually slowed things down further.
I couldn’t use the UserAgent to detect them either, as the useragent they are spoofing is the most common type for the most common browser (the lastest version of Chrome).
How I finally solved this and stopped the botnet dead
The attack pattern they were using was to attack our /my-account/ page with GET and POST requests. Both of these were passing email+password to try and discover a usable login. Cookies are required on our site in order to log in, so the pattern I decided to use was to check for specific cookies.
At first I added rules to nginx so that this traffic could be killed, by returning a 444 error, whilst using minimal server resources. But the attacker just moved onto a different IP address. The ultimate solution was to send them a huge file, if I suspected they were attackers:
- Check for cookie
- If cookie doesn’t exist, redirect to other URL
- That URL sets a cookie then redirects back to /my-account/
- If it’s their 2nd time on /my-account/ and still no cookie set, redirect them to download the biggest file I could find
Why it’s important to send them a massively large file
Running a hack like this uses server resources from the hackers computer or botnet. They clearly have resources, and I needed to slow them down. If I just ban the IP address, they’d move on to another IP address, so – instead – I send them a 120GB file.
That file will take at least a few hours for them to download, and given that they were running multiple threads, it holds up their attack for at least a day between requests.
The cost of downloading these massive files will cost them a lot of money through their ISP or hosting and potentially I may even fill up their hard disk (on their PC or server) and crash their script.
There are a few alternatives to sending them massive files, like limiting the number of login attempts per minute or banning their IP address but both of these are circumvented by using a rotating set of IP addresses.
This hacker had upwards of 300,000 IP addresses, so presumably is paying a service to use proxies around the world.
Another alternative would be to try to hack the hacker, but there are many problems with this. First, if you get a false-positive, you could end up hacking one of your own customers. Secondly, it’s illegal.
Redirecting them to a massive file on the other hand is harmless for false-positive real-users – they’ll just cancel the file download – and still has the biggest impact on slowing down the hackers. By my estimate, they had 10 threads running attacking my website – all of these threads quickly starting downloading a 120GB file which uses up all their bandwidth, hopefully gets them kicked from their hosting, and potentially even fills up the hard disk of their PC or server and crashes the script completely. Thankfully, none of this is illegal for me to do. I’m doing the equivalent of a URL shortener and redirecting them to another URL. It just so happens, if robots are visiting this, they’re going to flood their bandwidth and hard disk.
Pretty much immediately upon implementing this solution on my server, the attacks stopped.
Summary of the attack
Here is a summary of the attack that we underwent. The primary page they were attacking was our /my-account/ page. They were attempting both GET and POST requests against this page.
Once I had installed the WP fail2ban plugin, I enabled the option to track ‘missing user’ login attempts and found that the hackers have hold of a massive database of email addresses + passwords.
The WP fail2ban plugin logs unauthorised logins to /var/log/auth.log and then the same plugin provides you with a fail2ban script to ban these IP addresses if you wish (you can configure how many retries you’ll allow). That was useful in order to uncover the issue, but it didn’t help with actually combating the attacker in this case, because of the huge number of IP addresses they were using.
In the redacted log listed above, you can see the volume of attempts. For logins, they were attempting 4 or 5 times per second. This was outnumbered by the number of actual requests we were receiving (about 12 per second from the would-be hackers), but these login attempts use the full PHP, Nginx and MySQL stack to serve up the answer. You can’t cache authentication requests.
On the day of the solution above being implemented, my traffic recorded in Google Analytics changed radically.
- Page views – Up 148%
- Unique page views – Up 125%
- Bounce rate – Down 21%
On the bigger plus-side, sales for our plugins have resumed. These guys were using so much resource that, even when I doubled the RAM to the server, they would still consume everything I had available. That affects people viewing uncached pages the most. Your users using uncached pages are normally your most valuable, since they’re typically adding products to their cart.
If you ever find yourself under attack from a high-resource hacker, consider writing some conditional logic to send this type of traffic a massively huge file. It’s the only legal way I could find to slow and ultimately stop them. WordFence, Sucuri, fail2ban, they all rely on IP address banning. The user-agent of this hacker was of no use in identifying the traffic, so ultimately sending potentially dodgy traffic a huge file was the only way I could stop this attack.
For all users out there, be aware that this hacker has purchased a *huge* file of emails + passwords, so I highly recommend you install something like LastPass and use it to eliminate duplicate passwords from all the websites you use. By my estimates, their email + passwords list numbers in the 10s of millions, so get LastPass and get your online security sorted out.
I have a ‘friend invite’ link for LastPass that gets you 1 month Premium and gets me 1 month premium, although to be honest you don’t need premium, unless you have a team of people in your business.
Here’s the affiliate link:
And here’s the direct link:
If the hacker is out there listening, or if anyone else is interested in replicating this new behaviour to see how it works, just visit our site in incognito mode and then visit https://www.wpintense.com/my-account/. Just be sure to click cancel to avoid downloading the 120GB file.
And if the hacker is listening – try again mate, I’ll beat you again, no matter what you do. Feel free to comment if you’re brave enough.
- December Development Update - December 7, 2020
- Manual update for Super Speedy Search required - September 4, 2020
- How to enable debug.log to collect WordPress error messages - September 2, 2020