Server security with the Apache module ModEvasive

All websites down - what now?

The mysqld has gone down again - configuration errors and hardware defects can be ruled out after some searching. But what happened that all the websites hosted on the server went down because the mySQL server collapsed?

DDoS - Distributed Denial of Service, which means "widespread denial of service" that occurs when a huge number of requests cause an overload on the server. I don't want to go into the motivation of such an attack any further, but the fact is that the server must be able to withstand such unwanted attacks. Even if in our case the mySQL server spread its wings, the attack came via the Apache web server and this is also the appropriate place to defend against these attacks. Blocking unwanted requests in the web server ensures that they are not forwarded to other components of the "data network", such as the MySQL server.

Server security with the Apache module ModSecurity

Our first attempt to protect the server was the Apache module "ModSecurity". This is a WAF (Web Application Firewall) that guarantees access protection in real time. This is done via a large number of rules that can be extended or removed at any time. These rules check the HTTP traffic and reliably block unwanted requests. We installed the Open Web Application Security Project's freely available ModSecurity Core Rule Sets, often abbreviated to "OWASP CRS". It contains rules for the most common attack/threat categories, such as SQL injection, cross site scripting, file inclusion, php code injection, session fixation, etc. - Several hundred if not thousands of rules in total, each with its own ID and the option to remove the rule in the configuration, for example with "SecRuleRemoveById 981173".

After installing ModSecurity, however, I had a rude awakening:

• HTTP/2 calls can no longer be made, these must first be explicitly "allowed" in the ModSecurity configuration
• bots and other page requests that do not contain an "accept header" are blocked - the rules responsible for this must be identified and explicitly excluded
• Drupal AJAX calls are blocked, which is particularly fatal if updates, for example in Drupal views, are made via AJAX: these views are answered by ModSecurity with an "access denied 403"

The workaround, for example for the third case of AJAX calls, is to disable ModSecurity for the affected path by making the following entry in the configuration:

SecRule REQUEST_URI "/views/ajax" "phase:1,log,pass,id:99998,severity:1,msg:'Pass Drupal AJAX',ctl:ruleEngine=Off"

However, what happens if we have other Drupal modules in use that define their own AJAX paths? These will then also not work and this is the big problem with ModSecurity. There are reports of cases where a blogpost could not be saved under Drupal because the three strings "update", "set" and "=" were scattered in the content body text - ModSecurity recognized this as an SQL injection and prevented it from being saved!

So we started to compile a large list of "exceptions" from articles and tips available on the web in order to create our own ModSecurity rule configuration. However, we realized relatively quickly that under the premise of "choose your rules wisely", it becomes a full-time job, and every installation of a new Drupal module is an adventure, because you can never be sure that all paths defined by the module really work, or are not blocked by ModSecurity and one of its 1000 rules. This circumstance led us to an alternative option for Apache protection: ModEvasive.

The solution: ModEvasive with the configuration from HeliconTech

ModEvasive is an Apache module that initiates evasive action if DoS or brute force attacks are detected. The module tracks the incoming HTTP connections and checks how many requests have been made to a page within a configurable time frame. If the number of simultaneous requests exceeds a specified threshold, the request is blocked with a 403, thus saving a lot of bandwidth and server resources. The consequence for the attacker is that for the next few seconds (configurable) all requests from his IP end in a "403 forbidden", and the blocking time starts all over again with every new request.

By the way: a "normal surfer" who accesses via a proxy or a NAT gateway is not affected by the fact that other surfers with the same IP may be on the site at the same time - the module always counts "per listener" and an Apache always has several of these running at the same time.

Of course, this module, which operates at application level, also has its limits: if the attack is so massive that it utilizes the entire bandwidth or the server can no longer keep up with the hardware, the attack is still successful - in such cases, it is best to block the corresponding IPs completely, for example via .htaccess.

The configuration of ModEvasive is basically relatively self-explanatory, but as is so often the case, the devil is in the detail: The misleadingly named configuration parameter "DOSLogDir" specifies a directory in which the blocked IP addresses are stored as a file - this is required by ModEvasive to perform the "lock mechanism" for consecutive requests.

After the module was installed and configured, the whole thing was tested on the production server - of course there were surprises there too, because access to various pages was blocked.

After research, it turned out that one page, for example, contains a large number of small images (approx. 130) and each image represents a separate request - this naturally results in at least 130 requests being made to the same page from the same IP, whereupon ModEvasive, according to its configuration, correctly detects an attack and blocks the request. The same happened to the "bingbot", which was probably also "too aggressive" in crawling the pages.

The trick now is to configure ModEvasive in such a way that "normal" accesses are not yet considered an attack, but everything that goes beyond this is blocked. After many tests and multiple analyses of the log files, we have now adopted the ModEvasive configuration from HeliconTech and have been doing very well with it ever since: mysqld is doing its job reliably.