Thousands of requests from a thousand Automattic IPs for one Wordpress feed widget

A web site I manage has been frequently overloaded by thousands of the same requests from about a thousand Automattic IPs (192.0.64.0–192.0.127.255; Automattic owns Wordpress) for a feed requested via a Wordpress widget on a site hosted at Wordpress.com. The load can be several thousand identical requests per hour. Example request:

"GET /news/feed/ HTTP/1.1" 304 - "https://www.xyz.org/news/feed/" "WordPress.com; https://abc1.com" 

Most of the reply codes are 304 (not updated), so it doesn't actually cause the feed to be generated anew, but even so the load is often noticeable (and seems totally unnecessary). 

There is another Wordpress site with a widget for a feed, and it too makes a lot of requests, but from only one IP and not as many (which may of course simply reflect the popularity of the 2 sites as well as the hosting set-up). Example:

"GET /news/category/scotland/feed/ HTTP/1.1" 304 - "-" "WordPress/6.8.2; https://abc2.org" 

Curiously, the Wordpress.com requests include our site as the referer. They explained that as something the Simplepie feed parser, which they use, does, although the WordPress/6.8.2 agent doesn't do so.

There are a couple of other Automattic/Wordpress fetches that occasionally appear:

"GET /news/feed/ HTTP/1.1" 200 54932 "https://www.xyz.org/news/feed/" "Automattic Feed Fetcher 1.0"

"GET /news/feed/rss2/ HTTP/1.1" 301 - "https://www.xyz.org/news/feed/rss2/" "Automattic Feed Fetcher 1.0" (redirected to /news/feed/)

"GET /news/feed/ HTTP/1.1" 200 54932 "https://www.xyz.org/news/feed/" "wp.com feedbot/1.0 (+https://wp.com)" 

Another fetcher, Inoreader, adds not the feed URL but the domain root as referer:

"GET /news/feed/ HTTP/1.1" 304 - "https://www.xyz.org/" "Inoreader/1.0 (+http://www.inoreader.com/feed-fetcher; x subscribers; )"

Back to the original issue, after much discussion with Wordpress.com it’s clear that they don’t see any problem with their 1,000 IPs repeatedly making the same exact request. So the only answer was to increase caching, which does seem to have helped relieve the load.

First, I added this to the Apache config file:

<Location "/home/xyz/public_html/news/feed/">
    <IfModule mod_headers.c>
        Header unset ETag
        Header unset Vary
        Header append Vary: Accept-Encoding
        Header set Cache-Control "max-age=86400, public"
    </IfModule>
    FileETag None
</Location>
<Location "/home/xyz/public_html/news/category/scotland/feed/">
    <IfModule mod_headers.c>
        Header unset ETag
        Header unset Vary
        Header append Vary: Accept-Encoding
        Header set Cache-Control "max-age=86400, public"
    </IfModule>
    FileETag None
</Location>

(86,400 seconds is 1 day.) That probably duplicates the following, but I have them both:

<IfModule mod_headers.c>
    Header unset ETag
    Header unset Vary
    Header append Vary: Accept-Encoding
    <FilesMatch "\.(rss|txt|xml)$">
        Header set Cache-Control "max-age=86400, public"
    </FilesMatch>
</IfModule>

The following was already in the config file, but I increased the expiry times to match the above:

<IfModule mod_mime.c>
    AddType application/rss .rss
    AddType application/rss+xml .rss
</IfModule>
<IfModule mod_expires.c>
    ExpiresByType application/rss A86400
    ExpiresByType application/rss+xml A86400
</IfModule>

Finally, I increased the expiry time for the WP Super Cache plug-in to 86,400 as well.

See also:  Thousands of spurious requests to Mediawiki pages

A brief exchange seen on X regarding “the real” in Palestine

Rupert's Conscience @RupertsConscie1:

What struck me at #MarchforHumanity was the awesome diversity of Sydney. Friendly eye contact, nods, talk between people of every age, ancestry, gender, families, couples, friends etc interacting in harmony & happiness. The LNP have abandoned this 🇦🇺 They've abandoned the real 🇦🇺

El Rucio @ElRucioDos:

United in antisemitism. Beautiful.

Elizabeth Attard @ElizabethAttar5:

WAKE UP ANTI ZIONIST IS NOT ANTI SEMITISM - THE FEDERAL COURT OF AUST HAS MADE THIS LAW! STOP YOUR LYING. ISRAEL WHO ARE NOT EVEN SEMITES ARE THE WORST ANTI SEMITES AS THEY ARE MASS MURDERING THE REAL SEMITES - THE PALESTINIANS - TO STEAL ALL THEIR LANDS TO FORM "GREATER ISRAEL"

El Rucio @ElRucioDos:

Most Gazans came from elsewhere in the Ottoman Empire, eg, Bosnia and Circassia, as it shrank in the 19th century. Most Israelis came from Arab countries where they had lived for millennia and from which they were expelled in the 20th century.

Of note regarding "occupation", the region of Palestine was occupied by the Ottoman Empire for 400 years, and before that by several other empires. The kingdoms of Judah and Israel were established about 3200 years ago.

Further regarding occupation: How did this mosque in Jerusalam come to be built in the middle of Judean King Herod's temple?

See also: “A Brief History of Palestine Since World War I”