Learning Search

RSS for Post-Twitter News and Web Monitoring

With Twitter’s stability looking less and less assured, and other social networks making decisions so unpopular as to cause overt user rebellion, many Web users are wondering how they are going to keep up with news and current events. In a world where online entshittification seems to be accelerating rapidly, how can you track the news and updates you want without being tied to a platform which may at any moment decide to commit to full chaos muppetry?

The answer is RSS.

RSS feeds are available all over the Web. They’re text files for the most part so they’re easy to manipulate and process. And they’re a very old standard (I started working with them in 2000) so there’s an established infrastructure.  Best of all, RSS feeds allow you to consume and share information outside of the social media walled gardens.

(Did you know Facebook had RSS feeds at one point? I wrote an article about it in 2014. I don’t think it’s a coincidence that you can’t consume Facebook via RSS without third-party assistance these days; RSS gets you outside that walled garden!)

I could not do ResearchBuzz without RSS feeds. They’re invaluable. And I think if you learn more about them, you’ll appreciate why I consider RSS the most underrated tech on the Internet. That’s what this article is about: I’m going to explain what RSS feeds are, show you how to find them, go over some of the RSS feed readers available, and, finally, list several tools and resources you might find useful on your journey.

Let’s start with the basics: what is RSS?

What is RSS?

If you do a Web search for What does RSS stand for? You’ll get lots of different relevant answers, but for the purpose of this document let’s use Really Simple Syndication. RSS is a way of viewing content you’d normally view on a Web page, with the differences that the content is a) more structured and b) delivered to you.

What do I mean by structured? The data in RSS feeds is more organized and basic than the data in HTML pages. As an example, let’s look at the BBC’s home page on the Web:

A picture of the BBC's home page. There's a menu bar across the top, and the stories from BBC are represented by a series of squares with images and headlines superimposed over the images. It's understandable if cluttered.

This page makes sense to you, doesn’t it? You can easily see which headlines go with which descriptions and you can infer what to click on if you want to see an entire article. But what about when you look at the source code?

This is part of the source code from the BBC home page. There's tons of script references and metadata and site-related navigation stuff that's not useful to the average information-seeking site visitor.

Long before you see any kind of content in the source code for the BBC’s home page, you’ll see lots of script mentions and metadata and CSS. Even when you do get to the content, it’s not easy to quickly see what’s the link and what’s the description and what’s the title. You can unravel it if you examine the metadata for a moment, but it’s not immediately apparent.

Now, let’s take a look at the RSS feed for the BBC homepage:

This is a screenshot of the BBC's RSS feed. It's a little bit more formatted than you would usually find, but it's still plain, with a basic list of article headlines along with one-line descriptions.

That’s much simpler, isn’t it? It’s still as intuitive – you can easily tell the titles from the descriptions and which is the clickable link – but there are no images. There’s no navigation. There’s no additional details. There’s just a list of headlines and descriptions.

What about when you look at the source code?

This is a screenshot of the source code for the BBC's RSS feed. Unlike the BBC's home page source code, it is simple and contains only references to updated content on the BBC site.

Even the RSS source code is well-organized. That’s because RSS feeds are made of items, each of which has its own title, description, link, and publication date. Some feeds are slightly different in their content, but all RSS feeds are organized into discrete items with their own parts. That means RSS data is far more structured and manipulable than the data on a typical HTML Web page.

And that makes sense, if you think about it; Web pages have to have menus and disclaimers and advertising and season-specific content and content management controls. All an RSS feed has to have is information about recent content.

At the beginning of this article I said I could not do ResearchBuzz without RSS feeds. And that’s true, because I follow over a thousand RSS feeds which deliver information to me throughout the day. Do you think I could visit a thousand websites a day to check for new information? Even if I tried to visit a thousand a week that would be over 142 websites a day. Assuming it took me two minutes to visit a site and check for new content, I would spend over 4.5 hours a day just visiting websites.

Do you see why I’m so grateful for RSS?

Hopefully by this point I’ve given you a glimpse of why RSS feeds are so powerful and why I love them so much. But if they’re so great, you may be thinking, then why don’t I see them everywhere?

Well… you kinda do. You just don’t know it.

Finding Site-Based RSS Feeds

Just because you can’t see an RSS feed while you browse the Web doesn’t mean it’s not there. WordPress is arguably the biggest website platform there is, with 2021 figures estimating that over 40% of websites run on its CMS. Even if you don’t realize it, you probably visit lots of WordPress sites every day. (ResearchBuzz, ResearchBuzz Firehose, and Search Gizmos all run on WordPress.)

Did you know WordPress sites have built-in RSS feeds?

Let’s use the ResearchBuzz Firehose as an example. If you want to visit its website and read the latest articles, you’d go here:

RBFirehose.com

But if you wanted to get the RSS feed from the RBFirehose and read that in your feed reader (more about RSS feed readers later) you’d go here:

https://rbfirehose.com/feed/

If a site uses WordPress, you can check to see if it has an RSS feed by adding /feed/ to the end of the domain name. I don’t recommend doing that as a method to find all your RSS feeds because it would be extremely time-consuming. But if you come across some content you like and you want to check the site for an RSS feed, just add that /feed/ to the end of the domain name.

To test this I went to Google and searched for pickling blog. I got results for several blogs about pickling, as you might expect, including one called Food in Jars. This is what the home page looks like:

A screenshot of the Food in Jars home page. The top of the screenshot is blog stuff and the bottom part of the screenshot is nav stuff. In the middle of the screenshot is a FOLLOW me menu with several pointers to social networks.

Now, this isn’t a great site for a test – all I have to do is glance at this page to know that it has an RSS feed. That’s because it’s using the RSS feed icon:

Image5

on the  homepage under the FOLLOW ME section. That’s an excellent signal for the presence of an RSS feed, though the symbol isn’t used as often as it used to be. A word of caution: nowadays some websites are using the RSS icon to indicate when a podcast is available, which irritates me no end. In this case, however, it’s pretty clear that the icon stands for an RSS feed.

It’s also pretty clear that the website is a little outdated, as the FOLLOW ME section also links to Google+, which has been defunct since 2019. So I’ll double-check to see if this site really has an RSS feed by adding /feed/ to the end of foodinjars.com: foodinjars.com/feed/ .

And it does!

A screenshot of the Food In Jars RSS feed. It has been somewhat formatted (I suspect by a WordPress plugin) but it's still simpler than the Food in Jars home page.

This is way fancier than what an RSS feed normally looks like via a Web browser; I suspect it’s being formatted with a WordPress plugin. At the top of the page you’ll see that you have options to subscribe to the blog both by RSS and email.

I went to another search result, HildasKitchenBlog.com, and tried the /feed/ trick again. That site had an RSS feed too, but it wasn’t formatted. This is what it will usually look like when you find an RSS feed on the Web:

A screenshot of the Hilda's Kitchen Blog RSS feed. This feed has NOT been formatted by anything and basically looks like a big block of text.

If you want to check a site for RSS feeds and you think it might be a WordPress site, just add /feed/ to the end of the domain name. You might get a 404 error, but you also might get a page full of information! Let me give you a couple of other hints for finding RSS feeds.

Hints for Finding Hidden Site-Based RSS Feeds

When you’re checking a website for a default RSS feed, start at the root of the domain, not in a subdirectory. In other words, when I wanted to check Hilda’s Kitchen Blog for an RSS feed, I checked https://hildaskitchenblog.com/feed/ . I did not check https://hildaskitchenblog.com/recipe/st-johns-wort-oil/feed/ . That link works, by the way, but it provides a feed only for that item.

That’s another thing you have to watch out for – sometimes a site has a feed that’s not useful. A site might use a different RSS solution from the default, or it might have disabled the feed because it doesn’t want to syndicate content. You can’t just grab an RSS feed and assume it’s current or working.

I made you a couple of tools to help with this. The first one is a previewer that lets you look at an RSS feed without using a feed reader: https://rssgizmos.com/rssviewer.html . Enter the feed URL and it’ll show you the contents, basically formatting it so it’s easy for you to read.

A screenshot of the RSS Feed Previewer tool, showing the Hilda's Kitchen Blog feed in a much more readable format.

If you want to get more information about the feed in general, try the Feed Freshness Checker at https://rssgizmos.com/feedage.html . Enter up to 20 RSS feed URLs and the Checker will give you overview information about each one. Here’s the report for the Hilda’s Kitchen Blog feed:

A screenshot of results from the Feed Freshness Checker tool. Information provided by the tool includes how many items are in the feed, which is oldest, which is newest, and how many items on average are posted per day.

If you’ve only got so much bandwidth to keep up with RSS feeds, checking them in advance to see how active they are (or aren’t) before you commit to putting them in a feed reader will save you a lot of time in the long run.

Where to find Default RSS Feeds on Popular Web Platforms

Mastodon: Add .rss to the end of the user’s profile page URL: researchbuzz.masto.host/@researchbuzz.rss .

Medium: You have a couple of different options. Sometimes a Medium URL looks like this: wilw.medium.com . That’s a profile feed. In that case add add /feed/ to the end to get the RSS feed: https://wilw.medium.com/feed/ . If the URL format looks like this: medium.com/wordsthatmatter , that’s a blog feed. You have to put feed/ between medium.com/ and wordsthatmatter, like this: medium.com/feed/wordsthatmatter .

SquareSpace: SquareSpace sites do have default URLs but they’re kind of complicated; you can get all the details here: https://support.squarespace.com/hc/en-us/articles/215761717-Using-RSS-feeds .

Tumblr: Add /rss to the end of the domain name: https://www.designclever.co.uk/rss .

Wix: Add /blog-feed.xml to the end of the domain name: http://www.angelinadarrisaw.com/blog-feed.xml

WordPress: Add /feed/ to the end of the site’s domain name: rbfirehose.com/feed/

More Ways to Discover Site-Based RSS Feeds

As you can see, RSS feeds are everywhere. Unfortunately, there are not as many Web tools to find them as you might think. While writing this article I went looking for good RSS directories. I found some that were defunct and several that were overrun with spam (much like the rest of the Internet.) If you know of any good RSS directories, drop me a note. In the meantime, I’ve got two tool recommendations to help you get more RSS feeds – one I found and one I made.

Get RSS Feed – https://getrssfeed.com/ – Get RSS feed helps you find the RSS feeds hidden in websites. Just enter the domain name and Get RSS Feed shows you the RSS feed or reports when it isn’t available.

A screenshot of GetRSSFeed.com, showing an input menu and a successful output of an RSS feed.

WikiRSS – https://rssgizmos.com/wikirss.html –  A lot of information about Wikipedia entities is maintained through Wikidata, which manages over 11,000 possible data points for each entity. One of these Wikidata properties is “RSS Feed,” so  I made a quick search engine that keyword-searches Wikipedia and returns pages which have an RSS feed associated with them. Simple queries are your best bet with this tool.

A screenshot of WikiRSS. The screenshot shows a successful search for the query "Japan" and three out of many results.

So far in this article I’ve been talking about site-based RSS feeds – RSS resources that are associated with one website. CNN might have several different RSS feeds that are associated with different sections of its website, but in the end it’s all CNN content.

Site-based RSS feeds are the bedrock of your Web monitoring practice. They’re going to keep you up on the big-picture culture of whatever topics interest you. They’re going to teach you new topics and new vocabulary and new ideas. Site-based feeds are essential. But there’s another kind of RSS feed that’s less discussed. Sometimes this kind of feed is filled with content from one site, but more often its content comes from all over the Web. This kind of feed will teach you about new RSS so your feed collection doesn’t get stale.

I’m talking about keyword-based RSS feeds!

Keyword-Based RSS Feeds

Keyword-based feeds are exactly what they sound like – RSS feeds that are generated as the result of keyword searches, and not as representations of website or website sections (tags, categories, etc.) Usually keyword-based feeds aggregate content from many different websites, though sometimes they’re used to monitor keyword searches on a single website.

Here’s a source of keyword-based RSS feeds you’ll probably recognize: Google News! A regular search URL for Google News looks like this: https://news.google.com/search?q=cows while an RSS feed for a Google News search looks like this: https://news.google.com/rss/search?q=cows . Like a regular Google News search result, a Google News RSS feed aggregates articles from all over the Web. (Google Alerts, which monitor Google News, Web results, and other Google resources, are all available as RSS feeds.)

Keyword-based RSS feeds are useful in a couple of ways: first, since they’re generated based on the queries you choose, keyword-based RSS feeds can be very specific. If you’re careful about how you craft the query, you can create feeds that are almost 100% useful. Site-based feeds, on the other hand, usually have at least a little content in which you’re not interested. Second, keyword-based feeds will expose you to new sources of news and information. That’s important to keep the feeds you’re monitoring fresh – having a constant awareness of new RSS feeds makes it easier to ditch old sources when they stop publishing or otherwise become incompatible with your needs.

(Do you need some help crafting the perfect search for your keyword-based RSS feeds? I’ve written a few articles which can give you pointers: The Importance of Excluding Words When Setting Up Google Alerts, Tips for Angelina Jolie: How to Set Up a Google News Alert on a Famous Person (please note that Google’s location syntax no longer works), and In Praise of Inurl:. )

As you might imagine, finding keyword-based feeds is a little different from finding site-based feeds. I’ve got three ways you can bring more keyword-based feeds into your life.

Feedle –  https://feedle.world/ – Feedle is a search engine for RSS feed content. Moreover, you can get your search results as an RSS feed:

A screenshot of the search engine Feedle, showing the top three results for "agrivoltaic." An RSS symbol to the right indicates we can save this search as an RSS feed.

See that orange icon? If you clicked on it after running that Feedle search, you’dl get an RSS feed for the search agrivoltaic. The feed will update every time Feedle adds new content – you don’t have to visit Feedle again until you’re ready to run a new search.

Kebberfegg – https://rssgizmos.com/kebber.html – I painstakingly made the first version of Kebberfegg over fifteen years ago using Perl, and remade it much less painstakingly with JavaScript earlier this year. Kebberfegg stands for Keyword-Based RSS Feed Generator, and it’s is an easy way to make keyword-based RSS feeds for a dozen different sources, including Google News, Bing News, Reddit, Hacker News, and WordPress.

A screenshot of Kebberfegg. A query box up top lets you enter your query, while a series of checkboxes show your query as it would look as an RSS feed for a eleven sources.

CountryFeed – https://rssgizmos.com/countryfeed.html  – Several years ago Google News had a location: syntax that let you narrow your news search results by country or American state. That syntax doesn’t work anymore as far as I can tell. But Bing News still has a loc: syntax that lets you narrow your results by country, so I made CountryFeed. Enter the keywords you want to monitor and choose the countries for which you want to monitor that keyword. CountryFeed will deliver a set of country-specific feeds in an OPML file, all ready for you to import into your RSS feed reader.

A screenshot of CountryFeed showing the query box and LOTS of checkboxes representing countries worldwide.

Oh yes, RSS Feed Readers. I’ve spent most of this article talking about what RSS feeds are and how you find them, but we haven’t yet discussed how you use RSS feeds. It’s very easy: to read RSS feeds you need an RSS feed reader!

RSS Feed Readers

I started writing about the Web in the early 1990s. At that time a discussion of the Internet had to include an introduction that looked something like this:

The Web is made up of text files in a format called HTML, which is short for HyperText Markup Language. If you open an HTML formatted file with a text editor, it will be difficult for you to read because it contains a lot of formatting code meant to be used by a software program. To view HTML pages properly, you need a specific kind of program called a Web browser, or simply browser.

Almost 30 years later, that’s unnecessary – you know what the Web is and what HTML is. I find this kind of funny, because I have a very similar description of RSS feed readers:

RSS feeds are text files in a format called XML, which is short for Extensible Markup Language. If you open an RSS feed with a Web browser, it will be difficult for you to read because RSS feeds contain markup language that a Web browser generally can’t handle. To view RSS feeds properly in a way that allows you to save and manage them over time,  you need an RSS feed reader.

Unlike a Web browser which you use to move around the Internet, your RSS feed reader experience is built around the feeds you add to the reader. You can add them one at a time or import them all at once via a file format called OPML. Once you’ve added a set of RSS feeds to a reader, the software will automatically keep it updated, aggregating new articles from the feeds and adding them to your reader where you can read them at your leisure.  Most feed readers have features that allow you to save and share the content you find. Third party workflow apps like IFTTT can often be configured to work with feed readers.

Some feed readers are base-bones and basic – in fact, if you only want to follow a few RSS feeds, you might be able to make do with an extension that adds RSS feed capability to your Web browser. I recommend you use a dedicated Web app or client software, however – they’re much more feature-rich and useful than the browser extensions.

I’m going to tell you about two RSS feed readers I’m using right now, two I’ve used in the past, and some other options you have.

RSS Feed Readers I Use Now

NewsBlur – NewsBlur.com – NewsBlur is my main RSS feed reader; I’ve been a paying customer for years. I currently have 820 RSS feeds in my NewsBlur account and I read it several times a day.

The NewsBlur feed reader. A navigation list of all your RSS fees is on the left, while the content of the feed is formatted and presented on the right. In this case there's a story about food labels with a chicken giving you a judgy look.

This is what my setup looks like on NewsBlur. My list of RSS feeds are on the left. When I click I one, as I did with Purdue University’s feed in this screenshot, I see the content of that feed on the right. I’m using a list layout, but you can also view feed content in a number of other ways, including a magazine-type layout and a grid layout. The reason that the first item in this screenshot looks brighter than the other three is because it’s new since I last read the Texas A&M feed. The other articles are older items.

A different screen shot of NewsBlur, with the stories presented in a series of square panels instead of a list.

NewsBlur has a free tier for users with less than 64 feeds. For $36 a year you can have up to 1000 feeds and you get access to a few additional features. For $99 a year you get unlimited feeds and the content you read is archived and searchable. If you’d rather run your own version, the source code is available on GitHub.

Feedly – https://feedly.com/ – I use NewsBlur for all kinds of different RSS feeds. I use Feedly mostly for one kind of RSS feed – the country-specific RSS feeds I generate using Bing News’ loc: syntax.

A screenshot of Feedly. Unlike NewsBlur, all the feed stories here are presented in a series of square panels containing an image with the story title and description underneath. A small white x is at the top right of each story, so I can skim through them quickly, deleting as I read.

These feeds can be very active, and when a topic is hot they can produce a lot of near-duplicate content. Feedly has an AI-training feature that I can use to filter out a lot of the duplicates. I review my Feedly feeds in a magazine format because the articles don’t require deep reading – I can skim lots of headlines quickly this way.

Feedly is more expensive than NewsBlur and I’m not sure about the specifics of the free tier. The “Pro” tier is $6 a  month and allows up to 1000 feeds and several search/integration features. The “Pro+” tier, at $12 a month, increases the feed limit to 2500 and adds the AI filtering I mentioned before as well as a few other features.

RSS Feed Readers I Used To Use

Inoreader – Inoreader.com – I was once an enthusiastic user of Inoreader. It had a lot of features I appreciated, including the ability to get Facebook Pages as RSS feeds. Unfortunately I was reading a lot of Facebook Pages and when Inoreader changed its pricing to take that into account, I couldn’t afford it anymore.

I haven’t used Inoreader in a long time, but it looks like the pricing has changed to be friendlier to small setups and individual users.

RSSOwl – RSSOwl is an open-source RSS feed reader for Linux, Mac, and Windows. Unfortunately development stopped in 2019 and the developer is warning that it is insecure, so I won’t link to it here.

Other Feed Readers I Hear Good Things About

The Old Reader – https://theoldreader.com/

Feedbin – https://feedbin.com/

NetNewsWire – https://netnewswire.com/

BazQux Reader – https://bazqux.com/

MakeUseOf recently did a roundup article on RSS readers with AI features (in a good way).

RSS Tools

Earlier in this article I mentioned that RSS is a very old standard, and because of that it has an established ecosystem of tools. Because RSS feeds are usually text-only, there’s a lot you can do with them when it comes to extraction, analysis, and aggregation. To give you an idea of the possibilities, here’s a list of several RSS tools and resources that go beyond just reading. Some of them are mine. I’ve been busy.

RSS.app – A number of tools for creating and embedding RSS feeds.

Feed Validator – FeedValidator.org – If you’re having trouble with an RSS feed, you can check it here to see if it’s valid.

ChangeDetection.io – ChangeDetection.io – This is a website monitor which makes its page change alerts available as an RSS feed. I use this service to follow a number of libraries and government Websites via ChangeDetection’s RSS feed (which I read in NewsBlur.) This montior is packed with features and can have quite a learning curve but I’ll be honest – I haven’t done much beyond just dumping a list of pages in it and setting a daily check for each one. ChangeDetection.io costs $8.99 a month to monitor up to 5,000 URLs, which is an absolute steal.

WordPress Feed Comparison Tool – https://rssgizmos.com/wpfeeds.html – The main WordPress site can create two kinds of feeds – one for keywords and one for tags. This tool lets you create both at once and compare their contents.

RSStodon v2 – https://searchgizmos.com/rsstodon2/ – Generate hashtag-based RSS feeds across popular Mastodon instances, by language.

Bing News Query Checker – https://rssgizmos.com/querytest.html – Sometimes I want to set up a keyword-based RSS feed but I don’t know how well my query will work. BNQC lets you compare three different Bing News queries at a time to see what their RSS feeds look like – how busy they are, how many items they contain, etc.

OPML Maker – https://rssgizmos.com/opmlmaker.html – Adding RSS feeds one at a time to a feed reader can be tedious! OPML maker takes a list of RSS feed URLs and turns it into an OPML, which you can use to import all your favorite feeds into a feeder at once. If you’d rather explore an OPML file, the OPML Peeler presents you with a list of RSS feeds in an OPML file and makes them available as CSV text.

[Updates] Someone on Mastodon contacted me to let me know about https://openrss.org/ .  From the About page: “Open RSS is a nonprofit organization that provides free RSS feeds for websites and applications that don’t already provide them, so RSS feeds can continue to be a reliable way for internet users to stay up-to-date with content anywhere on the internet.”

3 replies »

    • Like resources for making your own when there are none available? I started getting into that and then I realized that was going to add another big section. Perhaps that can be its own article.

  1. Still love me some good old RSS stuff! Was one of my favorite classes to teach.

Leave a Reply