RSS

Parsing an RSS News Feed with a Bash Script

Jim Hall — Wed, 13 Jul 2016 16:11:11 +0000

I am involved in several free software projects, including one or two where I maintain the website. For one of those projects, we currently are updating the website. Ours is probably similar to other free software projects. We use a hosting service for several key services, including news, but we run our website on a web server that we own. In our case, we run most of our project on SourceForge and run the website on a third-party service, so the news and website are on different systems.

Not surprisingly, our project uses an RSS feed to pull news items from SourceForge to display on the project website. Previously, our website used a PHP script to cache a copy of the RSS file, then parse it every time a visitor arrived at our front page. That worked well, but was somewhat fragile if the remote news server was slow or temporarily unreachable.

As we work on the website update, however, I want to improve how we display news items. I want to automate the news service to pull down a copy of the RSS feed, parse it and save the news items to a local file that the website can just include. That reduces complexity on the website, and it means the front page will load faster every time.

Initially, I adapted our previous PHP website code into a PHP script that I could run from the command line. That did the job, but I thought there had to be a better way. I wondered if I could parse the RSS feed from a Bash script instead—and, it turns out that you can! It's easy!

A Brief RSS Primer

You may know that RSS is just a form of XML, which itself is a simple markup language. XML elements are values set off by tags. A tag is any markup that begins with the less-than symbol (<) and ends with the greater-than symbol (>). Tags are one of three types:

Start tags, to mark the start of a value (such as ).
End tags, to mark the end of a value ().
"Empty element" tags, which might contain values as attributes ().

In RSS feeds, most values are set off by start and end tags, like this link to Linux Journal:


http://www.linuxjournal.com/

RSS feeds have a well defined format. After some header information to describe the news website or "channel", the RSS feed displays separate news items inside ... elements. Each news item contains a title, link, publication date, full description and other fields. A sample RSS file might look like this:

Go to Full Article

Non-Linux FOSS: Vienna, Not Just for Sausages

Shawn Powers — Tue, 02 Jun 2015 19:51:59 +0000

by Shawn Powers

Although the technology itself has been around for a while, RSS is still the way most people consume Web content. When Google Reader was ended a few years back, there was a scramble to find the perfect alternative. You may remember my series of articles on Tiny Tiny RSS, Comma Feed and a handful of other Google Reader wannabes. I don't mention standalone RSS readers very often, however, because I don't like being tied to a single computer for reading Web sites. That's where syncing comes into play.

Vienna is an open-source RSS feed reader for OS X. Because it's written in Cocoa, it's available only for Macs. There are many alternatives for Linux and Windows, but the RSS reader options for OS X are surprisingly few.

The interface for Vienna is about like what you'd expect from an RSS reader. The view is customizable, and you can open complete stories in tabs to see the original Web site if you so desire. The real beauty of Vienna, however, is under the hood.

The Open Reader API is a protocol that aims to be vendor-neutral and completely open. Google Reader used to be the back end that everyone used for RSS feed syncing, and since its demise, people sort of re-invented the wheel in their own way. The Open Reader API is one solution that may catch on. It's already supported by BazQux and FeedHQ, and although adoption has been slow, hopefully it becomes the standard protocol for RSS syncing.

Luckily, if you're an OS X user, you can take advantage of the protocol right now with Vienna. Thanks to its great interface and open attitude, Vienna gets this month's Editors' Choice award. I think it's the first time we've given a non-Linux program the Editors' Choice honor, but its great interface and commitment to open standards makes us proud.

Go to Full Article

Command-Line Cloud: rss2email

Kyle Rankin — Thu, 22 May 2014 17:05:39 +0000

by Kyle Rankin

In my last article, I started a series called Command-Line Cloud. The intent of the series is to discuss how to use the cloud services we are faced with these days without resorting to a Web browser. I spend most of my time on the command line, so that's where I'd most like to interface with cloud services. My last article described how to use Google Calendar from the command line, and in this article, I talk about a more general cloud service—RSS feeds. If I had written this column a few months ago, it would have been more focused on replacing Google Reader itself, because that was the primary RSS aggregator I used, but Google preemptively killed off the service and left a lot of users, including myself, scrambling to find a replacement. Although a number of people were able to find some sort of Web-based replacement, I realized the main features I wanted (sorting stories by date and vi key bindings to view the next story) were absent in a lot of the existing Google Reader replacements. What's worse, a lot of people were using this as an opportunity to make a quick buck by selling access to RSS services (and of course, still capturing everyone's valuable Web-viewing habits).

I decided to take a completely different tack and convert my RSS feeds to e-mail in a special mailbox and use an interface I already was used to: e-mail on the command line using mutt. I decided to use the rss2email program, written by the great Aaron Swartz, to manage my feeds. This software pulls down RSS feeds and converts each story into its own e-mail message that it sends to you. This means you can use whatever e-mail program you want to read your feeds, but of course, because we are focusing on the command line here, I am going to talk about only mutt.

Installation and Configuration

The rss2email program already had Debian packages, so on my system, installing it was as easy as typing apt-get install rss2email. If for some reason it isn't packaged for your distribution, follow the steps on http://www.allthingsrss.com/rss2email/getting-started-with-rss2email to download and extract the rss2email tarball. This is Python software, so you will need Python 2.x on the system as well as some sort of local Sendmail program (Postfix or Exim works as well), or alternatively, you'll need to identify an outbound mail server you can use to send these e-mail messages.

Once rss2email is installed, you interface with it via the r2e command. To set up a new rss2email database containing your feeds, type:


$ r2e new youremail@yourdomain.net

Note that the e-mail address you use here will be the e-mail to which rss2email will send the e-mail messages. Once the database is set up, it's time to add feeds to it. You can do that with:

Go to Full Article

Beyond Google Reader: CommaFeed

Shawn Powers — Wed, 04 Sep 2013 20:44:25 +0000

by Shawn Powers

Now that Google Reader is officially gone, most folks have settled on a replacement of some sort. In fact, a few months ago I even went through the process of installing Tiny Tiny RSS as a viable and powerful replacement. At the time, there was only one feature I sorely missed, the "next unread blog" link. Approximately three days before Google Reader shut down for good, I found the holy grail of RSS readers: CommaFeed.

CommaFeed is an open-source project written in Java. It's offered as a free Web-based solution at https://www.commafeed.com. Although the interface is similar to Google Reader, it feels slightly stripped down. Thankfully, it provides a bookmarkable link that will take you to your next unread RSS entry. That feature is the single most important, and difficult to find, RSS reader feature I need.

CommaFeed unfortunately doesn't have a really good mobile interface, and it's lacking features in its Android app, but improvements are being made on both fronts. I must admit, however, that even more important than the "next unread blog" link feature, is the ability to download the https://github.com/Athou/commafeedsource code from GitHub and compile CommaFeed for self-hosting. I don't ever want to get "Google Reader'ed" again. Hosting is complicated, because it's a Java application, but the instructions on the GitHub site make it fairly painless. I recommend trying out https://www.commafeed.com before compiling and self-hosting, because CommaFeed's interface might not be for everyone.

Go to Full Article

The Google Giveth

Shawn Powers — Thu, 25 Jul 2013 20:33:32 +0000

by Shawn Powers

And the Google taketh away. So it is with Google Reader. A while back, Google discontinued its Google Wave product, because it never gained traction as a social-media platform. This surprised approximately zero people. More recently, Google announced it would be closing Google Reader on July 1, 2013. Far more people were surprised, myself included. In this article, I want to explore some options for those left in the lurch.

Those Clouds Look Ominous

I think even more interesting than Google eliminating Google Reader is the collateral damage it's doing to cloud computing in general. Reader is something I've used for years, depended on in fact, to keep up with the Web sites I find interesting. Google Reader is a program I'd happily pay for, but since it's free, I've always just counted my blessings and moved on. Now that it's disappearing, my dependence on free and/or cloud-based services is weighing heavily on me. Today it's Google Reader; will tomorrow be the end of Dropbox? Flickr? Google Mail?

Since Google's announcement regarding the demise of Reader, I've visited SourceForge and Github more frequently than I have in years. I don't like Google being able to affect my day-to-day computing so dramatically on a whim, and so I've been working hard to make myself less dependent on services like Google Reader. This is the first in what I expect might be a series of articles on self-sufficiency in this cloudy new world.

Web Shmeb

The simplest way to avoid losing your cloud-based solutions is to avoid Web-based services altogether. Before the original Web applications like Bloglines and Google Reader came about, people were perfectly happy with standalone RSS readers. Many folks still use a standalone application, and if you tend to browse the Web from the same computer all the time, a standalone application might be the perfect solution.

Liferea is a Linux-native application that does a nice job of managing RSS feeds. Like almost every other RSS application, it syncs with Google Reader, but thankfully, it also syncs with Tiny Tiny RSS (more on Tiny Tiny RSS later). Because it has the ability to sync with a back-end database, Liferea can provide the best of both worlds—namely, a local application for browsing RSS feeds, plus syncing with a common back end for reading on other devices and computers. Liferea has a simple interface, but if you want to burn through your RSS feeds, simple is good (Figure 1).

Figure 1. Liferea is simple, but that's not a bad thing.

Go to Full Article