Okay, an Ubuntu server I manage fell victim to a DDoS attack today. Usually this is unpleasant but not that big a deal. A few high server load moments & then it passes. Today was clearly different. For the record, I have years of server attack scars. But today felt different. Read on.
The solution was solved by re-enabling a ModSecurity
rule that prevents curl
access via a call to a pure IP address. Meaning if a hostname is mygreatsite.com
and the IP address is 123.456.78.90
and I have a virtual host for mygreatsite.com
& that is enabled. And ModSecurity
is active with the rule that prevents curl
access via a call to a pure IP address, then a visitor would get the proper content when they visit mygreatsite.com
, but are blocked by ModSecurity with a 403: Forbidden
if they attempt to access the site via 123.456.78.90
. Great problem solved!
But while the attack was happening, three things were clear:
- Apache Processes Through the Roof: The Apache process load as reflected in
Munin
was through the roof & consistently 3x higher than normal. - No Corresponding Web Traffic Logged: There was no corresponding traffic logged via Apache & reflected in AWStats. This was the case for a few hours. This is across all logs on all virtual hosts on the server including the default virtual host that would be connected to a bare IP address.
- Server Load Was Moderately High, But Not Typical Attack Level: While overall server load was high, it was only in the single digits—4, 5, 6, 7, etc…—and not a typical DDoS where loads can go double and even triple digit like 20, 30 & even 120 (!!!).
To me item number 3 was the most unusual part of this attack. I have not experienced a DDoS where server load was basically equivalent of a heavy traffic day & not an insane server load connected to the attack.
Additionally, running elinks
to fetch Apache server-status
from the command line showed the connection pool filling up like crazy within minutes of a restart:
elinks http://localhost/server-status?refresh=1
And past any of that, I just noticed an Ubuntu security patch today specifically for curl
issues. And this specific issue piqued my interest:
libcurl incorrectly validates wildcard SSL certificates containing literal IP addresses.
So here is the question: What the heck happened? I know re-enabling a ModSecurity
rule stopped the attack, but could it have in any way been connected to the libcurl
issue? And if so, how can I detect or prove that? And past that if the issue was with libcurl
—which I assume is coming from weaponized clients using the flaw—then does that mean that someone can simple keep a hacked older version of curl
around to DDoS again? And writing that, any hacker can just download source code & patch it to hack so why the flood today?
My guess is the patch to curl
made more people aware of the flaw & the “enthusiasm” of script kiddies around the globe brought a mad flood of scanning around within hours of the exploit being found? But again, this confuses me because it’s on the client curl
install, correct? I can’t imagine curl
on the server side would be victim to this.
0 Answers