The vast amount of information on the open web requires researchers and journalists to adapt with new tools to make sense of it all. Checking facts, gauging public opinion, and other lines of inquiry depend on being able to analyze the overwhelming amount of material posted online every day. Unfortunately, responding to the AI industry’s voracious appetite for data, companies stewarding this public data are looking to cash in and block access to the open web. This makes the web harder to navigate, and threatens all kinds of important research.

Reddit’s lawsuit against SerpAPI[1] and Perplexity threatens to do exactly that.

Reddit was once a very open platform, fostering a community of alternative apps and extensions putting users in control of their experience. Since the rise of AI however, it changed tack. Over the last three years Reddit has closed APIs to users, limited UI features, and otherwise restrained access without login. By making the voices on the platform harder to access, the company has positioned itself as a gatekeeper, and has since struck a lucrative data-sharing arrangement with Google.

SerpAPI, in contrast, is a tool which offers automated access to Google’s search engine results pages. It’s used by Perplexity, an AI-based search tool. Reddit sued to stop SerpAPI and Perplexity from bypassing Reddit’s own arrangement with Google. (Google brought a similar suit, which was dismissed by another court.)

It comes as little surprise that the legal weapon Reddit employed in this case is Section 1201 of the Digital Millennium Copyright Act, the same “anti-circumvention” law that has caused so much trouble for artists, innovators, and researchers over the years—and that EFF has fought against from the beginning. Reddit’s suit claimed that SerpAPI’s use of widely used techniques to access Google search results amounted to “circumventing” access controls on copyrighted works. 

There are major problems with this argument. 

First, Reddit doesn’t own the copyright in its users’ posts—the users do, and they grant Reddit the right to distribute the posted content freely.

Second, there’s no meaningful copyright infringement going on here. Perplexity functions like a search engine, responding to users’ queries with relevant links and optionally an AI-generated summary. Crawling and indexing the web to create a search engine was long ago ruled a fair use under copyright law.

And third, Google’s efforts to put rate limits on search queries and block some kinds of automated querying aren’t the sort of “access controls” that Section 1201 was written for. The law was passed in 1998 and targeted things like breaking the encryption on DVDs and other digital media. In contrast to physical or paywalled media, Reddit posts are visible to everyone, without a paywall and generally without a login. 

Unfortunately, a federal district judge ruled in July that the suit could proceed, concluding that Section 1201 applied. That’s a dangerous ruling, because it casts a cloud of legal uncertainty over lots of important research, including efforts to study public sentiment through social media and to preserve records of historical events. Without tools that can look at the web at scale, many kinds of research are simply impossible today. But if employing those tools risks Section 1201 liability, only the wealthiest institutions will be able to use them.

We cannot let the internet be rebuilt such that information is divided up among a handful of gatekeepers and sold back to us like cable subscription packages. If our knowledge of the online world comes only through wealthy institutions, the cost of knowledge will only increase and access to unbiased answers diminishes. 

[1] SerpAPI is an EFF donor. EFF donors do not influence our agenda or our positions.

Poke your head into just about any online social network—or any general conversations about internet culture—and you’ll likely find a boogieman: the algorithm. Since at least the moment Facebook introduced (and apologized for) its News Feed, “the algorithm” has been shorthand for the ways the tech giants control what we see and when we see it. In the age of enshittification, there is a push to reclaim our feeds and networks. Good news: there’s a tool that’s been around for decades that can help wrangle many of your feeds into something manageable: Really Simple Syndication, more commonly known as RSS.

What’s RSS and How Do I Use It?

RSS has been around since 1999, but its real publicity glow-up came from Google Reader, a newsreader service that Google offered between 2005 and 2013. Despite the alarm bells people rang at the time, the death of Google Reader wasn’t the death of RSS, and many replacements have come and gone over the years.

RSS may seem complicated, but it boils down to one general concept: when websites publish new content, like news articles, blog entries, webcomics, videos, or podcasts, that content gets added to an RSS feed, where your RSS reader (aka newsreader, feed reader, or aggregator) will show you that content in chronological order. If you’ve ever used a podcast player like Apple Podcasts or Spotify to follow different podcasts, you’ve used RSS. You can think of it like an internet-wide “follow” button, where you can track the contents of websites, users, and more.

People talk about RSS like it’s a power user’s secret trick to making the internet more usable, but the real secret is that it’s not that hard to set up and use. Here’s what you need to do:

• Find an RSS reader: RSS readers come in many forms. Feedly, NewsBlur, or The Old Reader, are web-based, but have their own apps (though they also support third-party apps). Others, like NetNewsWire, are app-based, and support either using a web-based RSS reader like Feedly, or a local file. Some live in browsers or web extensions. There’s an abundance of choice in RSS readers, and part of the fun is finding one that best accomplishes what you want to do. But don’t worry about finding the right RSS reader right away. One of the many magic tricks of RSS is that it is platform agnostic, and nearly every RSS reader—whether it's a website or an app, supports importing and exporting a list of the sites you subscribe to. This means you can change RSS readers in a couple minutes. If you need some help finding an RSS reader, Wired, The Verge, and Privacy Guides all have useful roundups. 
• Collect your feeds: As for adding websites to your feeds, the process is straightforward. Most RSS readers are designed to help find the feed for a site for you, so you don’t need to go hunting down a special link. Just drop the URL of what you want to follow in your reader, and if an RSS feed exists, it should be able to find it. If not, some sites, including ours (and our current podcast, EFFector as well as our last series, How to Fix the Internet), provide direct links to our RSS feeds.
• Sort, filter, and build your feed: Adding a bunch of new feeds can be overwhelming, particularly for news sites. RSS readers typically include folders, which let you group similar feeds together and can be great for lifting up low-traffic updates you don’t want to miss. Your reader may also have different filters, like the option to block any article that contains “sponsored post.” 

RSS Is the Best Way to Follow the News

It can be very difficult to follow the news, whether that means politics, tech policy, or your hobbies. Solutions like Google News or Apple News have tried to make this simpler, but many find that their algorithmic feeds are as often a source of frustration and annoyance as they are genuinely useful. And no matter how often you tap on news stories that matter to you from publications you respect, there may always be stories that refuse to bubble up.

RSS can make reading the news much easier, reliable, and more private. The vast majority of news sites have RSS feeds you can subscribe to, and many, including CNN, The New York Times, BBC, Wired, Politico, and many others, offer RSS for specific sections or special feeds that include the full text of articles for subscribers, so you aren’t just pummeled with a firehose of news all day long (we’ll get to a tip below in the next section that tackles this problem if they don’t have separate feeds, though). In many cases, you can read articles right in your RSS reader, never being forced to engage with wonky comments sections or poor design choices on websites.

Of course, the news isn’t just general news sites, it also includes hobbyist or more niche sites, local news offerings, and blogs. Most of these sorts of websites also offer RSS feeds, as do newsletter platforms like Substack or Ghost. 

RSS Offers One Way to Fix Some Social Feeds

Decentralized social media like Mastodon, Bluesky, and Threads, use RSS for user feeds, so you can follow your friend’s posts on Bluesky or Mastodon without actually having an account on either. This can be especially helpful for news sources, too—where you likely wouldn’t want to subscribe to a feed of everything a national news organization publishes because that would include dozens if not hundreds of stories a day, you can instead subscribe to their social media posts, which often get you the most breaking or important news.

The internet is more than just Facebook.

Some legacy social media works with RSS, too, including YouTube, Reddit (though that is currently at risk), and Tumblr. But others, like Facebook, LinkedIn, and Instagram, wall off posts behind account requirements that seem to pop up if you simply look at an account page for too long, let alone come in from an RSS feed. These walled gardens prevent information from getting out there, which ranges from annoying, like when your favorite local brewery only posts their food truck schedule on Instagram, to dangerous, like when local public services only post to a Facebook page.

The internet is more than just Facebook. It’s more than Mastodon or Bluesky, too. It’s a decentralized smorgasbord of websites, tools, feeds, newsletters, social profiles, and more, and treating it as such will help us wrangle the information we want and trust. 

Other Surprising Places You’ll Find RSS Feeds

When in doubt, try copying and pasting the URL for a site into your RSS reader of choice, you might be surprised to find a feed that proves useful to you. Many places on the internet may offer RSS feeds without you even realizing it. For example, if you want to keep an eye on an artist’s prints that you like, but you don’t have Instagram where they usually post, you might be able to subscribe to their webstore, as some shopping platforms, like Big Cartel, create XML feeds that some RSS readers can read. And for something even more tweakable, even Google Alerts can be turned into RSS feeds.

RSS is one of the best examples we have of the open web, where we can design and customize how we experience the internet, not the other way around.

If you prefer to track policy over products, then you’ll be happy to know that government sites often support RSS, including most U.S. government sites, many of which break them into different sections like the U.S. Department of State’s various feeds. Many local governments or other public services, like fire departments may offer the same. Some universities (and university newspapers) also sometimes offer some RSS feeds. 

And even if a website doesn’t have an RSS feed, there are workarounds from tools like RSSHub, RSS-Bridge, and RSS.app that require varying levels of technical expertise or a willingness to pay subscription fees.

RSS is one of the best examples we have of the open web, where we can design and customize how we experience the internet, not the other way around. RSS has come in and out of fashion, been declared dead, and has come back, every time. Open systems are the best way forward to a free, equitable internet, and the resilience and continued reinvention of RSS has shown just how creative the web community can be with open protocols.

Correction: clarified that Big Cartel creates an XML file, not an RSS feed.

The ability to access publicly available information using automated tools is a central value and benefit of a free and open internet. Automated access—often called crawling or scraping—powers important, useful tools for locating, preserving, and analyzing online information. For example, crawling and scraping helps journalists, researchers, and watchdog organizations report the news, find security flaws, and investigate discrimination. Crawling the web allows non-profits like the Internet Archive to preserve historical copies of websites. Tools for automated comparison shopping allow consumers to find the best deals on items they want to buy. And so on.

Yet the open internet access is increasingly under threat from publishers and Big Tech companies alike. Fearing lost advertising and licensing revenues, website operators increasingly claim that they need to lock down their sites from bots that crawl public web content to train or operate AI models. Some companies are even trying to embed their business models into internet standards by changing Internet Engineering Task Force (IETF) technical standards that shape much of the internet.

Many of their economic anxieties are understandable. AI bots can strain websites’ infrastructure, in some cases, degrading site performance or taking them offline altogether. Upgrading systems costs money that some sites may not have. And AI is likely to disrupt the business models many publishers adopted in response to the rise of the internet, if users rely on AI overviews instead of visiting source websites.

However reasonable these fears may be, the answer is not to change the IETF standards from neutral protocols that encourage openness to restrictive requirements designed to monetize internet access.

The worst of these proposed standards would give websites far greater ability to automatically block legitimate, lawful scraping and crawling. For example, the AI Preferences working group is working on proposals to give publishers a way to express “preference signals” against crawling web data for AI-related purposes, including to train models, generate outputs, and help users search the web. These preference signals would be expressed through robots.txt and could potentially become legally binding in some jurisdictions.

Another working group, called Web Bot Auth, is pursuing efforts to protect sites from overly-aggressive bots that strain website resources—a positive goal that could meaningfully improve the internet in the AI era. But Web Bot Auth is simultaneously pursuing a much more dangerous path as well: standards changes that would enable sites to cryptographically identify bots so that they can more easily block anyone they wish—not just “bad” actors, but competitors, dissidents, or anyone who hasn’t paid for the right to access sites using automated tools. If sites restrict crawling to a preapproved list of cryptographically authenticated bots, they could require licensing payments from those wishing to crawl their sites. This would close off the open web to researchers, archivists, and startups without the ability to pay for automated access.  

Websites may have legitimate reasons to worry about AI’s impacts on their traffic and advertising revenue, but those reasons must be weighed against the benefits of the open web. These proposals would effectively give website operators veto power over a wide range of important uses—from the investigations and archival works described above to accessibility tools for people with disabilities, to research efforts aimed at holding governments accountable.

That is why we are fighting back against these threats to open access. EFF and our allies in the open internet community have successfully resisted some of the most dangerous IETF proposals thus far—and won’t stop working to protect the open web from efforts to manipulate internet standards to undermine the right to freely access the internet in any legal way, including with automated tools.