---
title: "When Bots Knew Your Architecture Better Than Your WAF Did"
url: https://www.axelerant.com/blog/drupal-facets-ddos-mitigation
published: 2026-07-28T14:30:00Z
author: "Ulka Neil"
source: Axelerant Thinking
---

# When Bots Knew Your Architecture Better Than Your WAF Did

> A distributed bot attack hit a Drupal Facets endpoint at 1,680 requests per minute from 192,705 IPs across 24 hours. Here is the forensic response, the WAF gap it exposed, and the layered CDN mitigation that stopped it.

At 05:30 UTC on March 6, 2026, the first requests began arriving at a global enterprise website. Nothing alarming, a few hundred hits to the /search-result endpoint. Normal enough for a high-traffic Drupal site serving millions of visitors across multiple continents.

By 09:10, the traffic had changed character. By 13:00, it was a flood.

By the absolute peak at 16:39 UTC, 1,680 requests were arriving every single minute, not from a handful of attacking servers, but from an estimated 28,141 unique IP addresses in that hour alone. All 16 nginx proxy nodes were saturating. Backend application errors were piling up. The Solr search cluster was under load it was never designed to sustain. The site was degrading. And the client's CTO was asking questions.

In the 24 hours that followed, this attack would generate 278,751 requests to a single endpoint. It would represent 70.9% of all traffic the site received that day. And it would reveal something important: the most dangerous bot attacks today are not the loud, blunt-force floods of the past. They are quiet, precise, and architecturally informed.

This is the story of what happened, how the engineering team responded, and what every Drupal team running faceted search should understand before the same pattern finds their site.

## The Target Was Not Random

The client is a global enterprise technology company with a multi-region, multi-locale [Drupal](/platforms/drupal) site hosted on Pantheon. The site uses Apache Solr for full-text search, with Drupal Facets providing filterable content discovery across product categories, publication dates, content types, and more.

That combination, Drupal Views feeding Solr, surfaced through Facets, with filter parameters passed in the URL, is extremely common across enterprise Drupal deployments. And it was, in this case, the precise target.

The attacker's approach was methodical. Every request to /search-result carried exactly four Drupal Facets filter parameters. A typical request looked like this:

> `GET /search-result?f[0]=authored_on:2021-04&f[1]=authored_on:2024-07&f[2]=product:649&f[3]=blog_type:cybersecurity`

This was not random noise. The bots were systematically enumerating every month from December 2015 through March 2026, combining date filters with product IDs, content types, and solution categories. The theoretical permutation space ran into the hundreds of millions. The bots were building a catalogue, and breaking the site in the process.

## Why Every Caching Layer Failed

To understand why this attack was so effective, you have to understand what happens when a Drupal Facets request cannot be served from cache.

A standard Drupal page request has a clear, predictable cache path: Pantheon's AGCDN, then Varnish, then Redis, and only on a cache miss does the request reach Drupal and the database. A well-configured site serves the vast majority of traffic from the CDN edge without touching the application at all.

Faceted search breaks this model. Each unique combination of filter parameters produces a unique cache key. With authored_on dates spanning ten years and product IDs in the hundreds, there is no practical way to pre-warm the cache against every permutation. Each bot request was, by design, a cache miss.

That means every request traveled the full stack:

> `CDN -> Varnish -> Redis -> Drupal Views -> Solr (full-text search + facet aggregation) -> Drupal render -> Response`

The Solr step is the expensive one. Facet aggregation is computationally intensive. At 1,680 requests per minute, the backend was being asked to run over 1,600 simultaneous complex Solr queries, each one unique, each one non-cacheable. The nginx upstream connection pool had no chance.

The resulting error was diagnostic:

> `[alert] worker_connections are not enough while connecting to upstream
upstream: "fastcgi://unix:/shared-run/php-fpm.sock"`

Over the 24-hour window, this error appeared 5,932 times, accounting for the 5,742 HTTP 500 responses real users encountered.

## The Defense That Should Have Worked, And Why It Did Not

There was already a WAF rule in place. It applied a global rate limit of 30 requests per minute on the /search-result path:

> `key=host_path | limit=30 | delta=1 | window=1 | duration=1m`

On paper, this rule blocked 222,306 requests, 79.7% of all bot traffic. An 88% effective block rate sounds strong. But the framing masks a structural problem.

The rule keyed on host_path. It counted every request to the /search-result path against the same shared counter. When the counter hit 30 per minute, the rule fired, but it fired globally, meaning the first 30 requests in each 60-second window passed through regardless of their source. During peak minutes, those 30 slots were taken by bots. Legitimate users arrived as request 31 and received a 429.

Worse, the rule had no awareness of query parameter count. It could not distinguish a single-filter search from a four-filter bot request. The existing Facet Bot Blocker module was configured to block requests with more than two parameters, but log analysis revealed a critical gap: 31,431 three-parameter requests returned zero WAF blocks, with 23,072 receiving HTTP 200 responses. The bots were actively exploiting the edge of the configured threshold.

The attack was operating with precise knowledge of where the defenses sat.

## The Other Reason IP Blocking Failed

The attacking botnet used 192,705 unique IP addresses across the 24-hour window. The average was 1.45 requests per IP. During the peak minute, the ratio of unique IPs to total requests approached 0.95, meaning nearly every request came from a different address.

Per-IP rate limiting is architecturally blind to this pattern. There is no individual source to block. The attack's destructive power came from width, not depth.

The user-agent picture was equally sophisticated. The bots rotated through two tiers of spoofed user-agents: a pool of plausible-looking Chrome versions (indistinguishable from legitimate traffic) and a set of technically impossible combinations, "Windows Server 2012 R2 Standard" in a browser UA, or CentOS and Ubuntu listed simultaneously, that revealed the scraper runtime while evading simple string-match rules.

The tell-tale fingerprint was not the IP address. It was not the user-agent. It was the query parameter count.

## The Response: Forensics First, Then Layers

The Axelerant engineering team's first move was not to patch, it was to understand. Deep log forensics on nginx-access.log and nginx-error.log from the incident window produced the full picture: the attack timeline, the three wave pattern consistent with scheduled botnet job rotation, the IP distribution confirming the distributed proxy pool, and the 4-parameter signature that was both the attack's mechanism and its most reliable detection signal.

From that analysis, the mitigation approach was layered from the edge inward.

### Layer 1: CDN VCL Hard Block

A Fastly VCL rule targeting the four-parameter request signature was deployed at the CDN edge before any request reached the origin. The logic was precise:

> `if (var.param_count >= 4) { error 403 "Forbidden"; }`

This single rule had zero false positives against legitimate users, no real user selects four simultaneous date-range filters in a single search. It addressed 87% of attack volume at the infrastructure layer, before Drupal, before Solr, before any backend cost was incurred.

A second layer added rate-limiting specifically for three-parameter requests on a per-IP basis, closing the WAF bypass gap that had been allowing 23,000 successful bot responses per day.

A third rule blocked fabricated user-agents using technically impossible OS/browser combinations, immediately containing the persistent high-volume bot control nodes identified in the log analysis.

### Layer 2: Application-Level Tuning

The Drupal Facet Bot Blocker module threshold was reviewed and aligned with real user behavior data. Monitoring cadence via Pantheon metrics was increased to three times weekly, creating a feedback loop between traffic pattern data and rule refinement.

### Layer 3: Edge Architecture Expansion

After evaluating Cloudflare, the team expanded the previously deployed Wangsu CDN globally as the primary edge layer in front of Pantheon, adding a first line of defense that could apply security rules before traffic reached the hosting stack at all.

### Layer 4: Structural Fixes and Form Hardening

Parallel work addressed the broader attack surface: vulnerable high-traffic forms were removed or replaced with math CAPTCHA (blocking approximately 95% of automated submissions), legacy reCAPTCHA v3 scripts were removed from decommissioned form templates to improve frontend performance, and a recurring Cloudflare review cadence was established as an ongoing [quality engineering](/what-we-do/engineering/quality) practice.

## What "Fully Mitigated" Actually Looks Like

The outcome, confirmed in the team retrospective months later: the DDoS attack was fully mitigated. 25 million bot hits per month were being contained by the combined Cloudflare and WAF rule hardening. Zero recurrence. Site availability restored. Backend error rate returned to baseline. Uptime monitoring data clean across all subsequent reporting periods.

The RCA was published internally and shared with the client. The VCL ruleset was documented. The incident became the baseline for a long-term security posture, not just a one-time fix.

## What This Means for Your Drupal Team

The Facets endpoint was targeted because it was the most permutation-rich, cache-defeating, computationally expensive endpoint on the site. But the underlying vulnerability is not unique to Facets. Any dynamic endpoint that generates unique cache keys per request, paginated content lists, filtered blog archives, search with query parameters, API endpoints with arbitrary filters, is a candidate target using the same attack model.

Reducing the number of available facet filters is a sensible hygiene measure. It shrinks the permutation space and limits data exposure if scraping succeeds. But it does not prevent a distributed botnet from targeting whatever dynamic endpoints remain. The defense must be architectural.

The practical takeaways for enterprise Drupal teams:

### Rate limit by request signature, not just by IP or path

Query parameter count is a reliable, low-false-positive signal for this class of attack. A hard CDN block on requests exceeding a defined parameter threshold can eliminate the majority of attack volume before it reaches your origin.

### Separate your rate limit counters

A global shared counter that fires after 30 requests per minute penalizes legitimate users when bots exhaust the budget first. Per-IP rate limiting on borderline request patterns is more precise.

### Monitor for the 1:1 IP-to-request ratio

When unique IP count per minute approaches total request count, you are seeing a distributed proxy pool, not organic traffic. Standard per-IP blocking will not help. Request signature rules will.

### Build observability before you need it

The attack was underway for hours before the full impact was felt. Real-time per-path request rate alerting, combined with 5xx error rate monitoring and nginx connection pool metrics, would have triggered incident response in minutes, not hours.

### Treat the WAF bypass as a feature of modern bot attacks

Sophisticated botnets probe for threshold edges and operate just below them. Review your rules regularly with actual attack log data, not just theoretical thresholds.

## Security at the Intersection of Architecture and Delivery

What stands out about this incident is not just the technical response, it is the discipline of the process. Forensics before patching. Layered controls rather than a single fix. Documentation that captured not just what was done but why, preserving institutional knowledge for the next incident and the next team.

The best security response is one that converts a crisis into a capability. That is what happened here: a sophisticated attack became a documented, reusable, and continuously monitored security posture.

If your Drupal platform runs faceted search, dynamic content discovery, or any endpoint where query parameters reach the backend uncached, this pattern is worth pressure-testing before it finds you.

## Related reading

For a related Drupal engineering deep-dive, see [When WYSIWYG lies: debugging Drupal's image derivative race condition in CKEditor](/blog/drupal-ckeditor-image-derivative-race-condition). For an architectural take on cacheability, identity, and upstream latency in a Drupal 11 build, read [How to build a Drupal member portal on an AMS you cannot replace](/blog/drupal-member-portal-ams-architecture). More on how we approach [platform engineering](/what-we-do/engineering/platform) and the [Drupal platform practice](/platforms/drupal).

[Axelerant](/what-we-do/engineering) helps enterprise digital teams build, harden, and scale [Drupal platforms](/platforms/drupal) with the engineering depth and delivery discipline that complex environments demand. If this pattern sounds familiar, [get in touch](/contact).

---

Read on the web: https://www.axelerant.com/blog/drupal-facets-ddos-mitigation