Botnet Attack – RunCloud CTO Statement

We are very sorry that our users have to suffer these problems one by one. Right after architecture migration, which took longer than we expected (almost 18 hours), we have been attacked by a botnet. What is a botnet exactly? A botnet is a large number of computer, IoT and servers that have been hacked. Once the devices have been taken over, it will be known as bot or zombie. Thus, a botnet is a network of zombies or bots that can be controlled remotely by the attacker to do their dirty works.

The Attack Vector

What happened at RunCloud last night (7PM – 2AM GMT+8) was an attack to brute-force email and password. We were down from 7PM until 1:30AM. And 1:30AM – 2AM was a restoration process. We were attacked by 7000+ devices at the same time trying to test email and password combinations. We know this because when the incident happened, we piped all those authentication attempts to a text file in order to see all those tested emails and password. Those passwords were unique to the email. When we checked the tested email, the emails were not found inside our database. So whoever they are, they were testing from an unknown leaked database (dictionary-based attack) and not brute-forcing password one by one.

We built our panel using Laravel and Laravel has very good security to prevent cross-site requests by providing a CSRF token. When we analyze the log, we found that those bots load the page to get the CSRF token and then submit those tokens to the authentication page. Whoever using Laravel, you are not able to mitigate from this attack inside Laravel itself. But you need special equipment in front of that Laravel to prevent this. This is not Laravel’s fault, but that person who creates the attack script can attack any range of framework. It doesn’t matter if you are using NodeJS, Python, Java, Go or even C itself, you can’t run from this.

How We Were Holding

During those attack, our servers were holding just fine, however, we have maxed out the PHP process and thus creating 504 gateway timeout and 502 gateway error by the load balancer. We were thinking about upgrading the server PHP process, but that will create more of a window of opportunity for the attacker to attack.

By reading the webserver log, we were able to distinguish the difference between a true user and a zombie, but we will not be revealing this information. Using the information, we were trying to create middleware to prevent those bots from creating another attack by sending them 404 response. However, those bots do not stop until they found 302, 301 or 200 code to send those valid data back to the attacker. Besides the middleware, we already placed throttle limit when doing authentication, but those bots didn’t honor 429 HTTP code so they keep trying although our servers have sent them 429 code.

Mitigation Plan

If anyone ever runs a DNS query to our domain (runcloud.io), they will find that we were using Digital Ocean DNS. Because of the attack, we moved our DNS to Cloudflare since we are no longer able to withstand the heat coming from those attack. Inside Cloudflare, we use Page Rules to set our authentication page is under attack. Cloudflare did a good job at preventing it. But to prevent those additional attacks, we are using other tools from Cloudflare. Cloudflare successfully blocked those attacks and the last attack happened at 01:25:54AM. We were not using Cloudflare before because we were hoping that by horizontally scaling our server, so we can withstand those attacks. But we had not done it because would create a window of opportunity for the attacker to attack.

The IP Addresses Lists

We have successfully collected the IP Addresses of the bots and please check whether your server, computer, network or IoT devices have been compromised by the attacker to create a botnet. This list consists of 7379 unique IP Addresses that we have piped to sort | uniq to sort and get a unique IP Address. This is the IP Address that we have collected ourself and not including the address blocked by Cloudflare. So we don’t know the exact infected devices.  http://pasted.co/a925f9ba

Share This On
Share on facebook
Share on twitter
Share on linkedin
Share on reddit

4 thoughts on “Botnet Attack – RunCloud CTO Statement”

  1. “But to prevent those additional attacks, we are using other tools from Cloudflare”

    -What are those “Other Tools?”

    -And are you on a “business plan” at CloudFlare?

    1. Hey Terry, Cloudflare offers quite a bit of protection and additional tools that a personal plan wouldn’t normally have, so we have enabled everything on the business plan that would further protect us.

Leave a Comment

Your email address will not be published. Required fields are marked *

You May Also Like

Scroll to Top