FSF SysOps cleaning up the Internet

The SysOps team, consisting of only two full-time FSF staff members and a handful of volunteers, spends a considerable amount of time preventing attacks from taking down our services in addition to our usual operations. We self-host over sixty-three different services, platforms, and websites for the FSF staff, the GNU Project, community projects, and the wider free software community on our own twelve-year-old hardware which uses free BIOSes. Like for many other entities on the Internet, attacks often come in the form of a Distributed Denial of Service attack (DDoS), which can use thousands of devices to fulfill its goal of flooding its target's servers, and ultimately bring it down.

Some of our most serious incidents happened in 2016 with a third-party DDoS mitigation service, and even involved the FBI. I am not going to write about the specifics of our recent or ongoing incidents because I do not want to help anyone that is actively attacking us at this moment. What I can mention however, is that one of the recent attacks from the last few months, required blocking more than 40,000 IP addresses from a DDoS attack. And I can also share some of the tools that we use weekly to investigate attacks, which are somewhat simplified for this article but hopefully helpful for other sysadmins:

Monitoring

With tools such as Prometheus and Uptime Kuma, we can be alerted when a service goes down or when the response time increases. A pattern of alerts may indicate that an investigation needs to take place.

Analyzing logs

The logs of the service affected usually tell a story, but it is hard to read the logs by going line by line. Logs can be analyzed using variations of classic utilities such as Bash, cat, zcat, grep, egrep, awk, sed, sort, uniq, and tail. I typically find the top ten IP addresses communicating with a service and then look for the top three user agents used by that address and a sampling of what they are accessing.

Finding patterns

You will probably notice odd traffic looking at log files from any publicly available website. If a normal crawler is accessing 1,000 pages, something is likely off when another address has 30,000+ lines in the log. Real examples of when something is off include: continually looking for WordPress specific pages and the site is not using WordPress; claiming to be Googlebot and their host does not resolve as Google; not following the robots.txt file; if their user agent is a standard desktop browser and they are generating thousands of lines of log; if a user agent string is empty; or if they request pages several times a second.

Looking up ASNs

We use IPtoASN to provide updated autonomous system number (ASN) tables at no cost. It is the only one we have found that involves downloading a database locally which can be queried. ASN is an identification number representing a collection of addresses that belong to a network operator. Once we have a local ASN table, we query the table with the addresses found to be questionable to gain a better understanding on a macro scale. Adding this information, as well as the behavior to firewall rules, helps us look for patterns over time.

Blocking

We use a variety of firewalls to prevent addresses from reaching our servers before restarting the affected service. We may need to block individual addresses, CIDR addresses, VPS providers, or even entire ASNs. When blocking an address or a block of addresses is not enough, rules can be written to look for patterns of behavior with tools such as fail2ban. The default example rule when fail2ban is installed is to temporarily block an IP address after three failed SSH attempts. Fail2ban rules can be written to match patterns in any log file. When writing fail2ban rules, be aware that it can be easy to write rules that are too aggressive, which can cause a denial of service. While the fail2ban defaults are sensible, you could watch the fail2ban logs carefully and adjust the settings to avoid blocking valid traffic. It is important to find a good balance with fail2ban rules.

Abuse reports

Sometimes blocking isn't enough to prevent or stop persistent abuse. This abuse can be reported to hosting companies and Internet Service Providers (ISPs). Sometimes IPS's provide a page to report abuse, but that page may use nonfree JavaScript. We can usually get around using nonfree JavaScript in cases like these by sending an email with a description of the abuse, a snippet of the log, and expected behavior to abuse@theirdomain.com. Many abuse reports go unanswered, but when they do answer, it can be worthwhile.

We see three kinds of abuse activity most often on our web servers. Several organizations dedicate themselves to vulnerability scanning, an abuse that actively scans the Internet all of the time, most of which can be blocked with firewalls at the router level. Another kind of scanner that can be persistent are programs written to search through entire websites, or website crawlers. Most crawler programs are gentle on the servers that they scan, but the same cannot be said for ones that ignore the robots.txt files, scan too fast, and take down sites, especially those written by large language model companies. We also see some continuous integration automation, which constantly scans websites for changes, such as those to our licensing pages.

Different technology

For various reasons, Internet traffic coming in and out of some locations, such as the People’s Republic of China, behaves different than anywhere else in the world. This could have to do with the Great Firewall of China, carrier-grade NAT, and routing. Many different behaviors can be witnessed simultaneously coming from the same IP addresses that link to Chinese ASNs. Because there is good traffic mixed with bad traffic coming from the same addresses, traditional tools such as fail2ban and firewalls break functionality. If I create a fail2ban rule to block nefarious behavior seen from a Chinese address, I will also be blocking the good traffic that is trying to use the same IP address. I have recently started using a web application firewall called Modsecurity, which allows traffic sculpting by user agent instead of just by IP address. Currently, we are just rate-limiting some user agents, but we plan to add more configurations in the future.

What can you do to help?

Shepherding our minimalist resources, conserving our computing, and stretching our financial resources take a lot of time and effort. Even with all of the tools we use to investigate and block attacks, we are always looking for more help, and we list many ways to volunteer at volunteer>. The best way to signal long-term support with the FSF SysOps team, and the FSF as a whole, is to become an FSF associate member.

Hacker. © 2017 by AUIC Official. This image is licensed under a Creative Commons Attribution 2.0 Generic license.

Copyright © 2024 Free Software Foundation, Inc. This article is individually licensed under the Creative Commons Attribution-ShareAlike 4.0 International license.