Although I perhaps should have been using the time in other ways, I’ve spent considerable time and effort building a custom news aggregator. I was able to leverage SimplePie and PHP to do a lot of the heavy lifting, but I also wanted to scour the Internet, social media, and all the trendy Web 2.0 networks for the best content related to the hobby of miniature painting.
I learned a few things along the way, like not all feeds are created equal, and not everyone uses hashtags or even content management systems to their fullest. Some preferences/prejudices were reinforced and some new tricks and tools were discovered, so since I’m still unemployed and you can use these tips and tools to monitor just about anything, I thought I’d list my Top 10 sources/techniques you can use to get RSS or Atom feeds.
- Blogs It goes without saying there are a lot of blogs online. Blogger/blogspot.com and WordPress seem to be the two most popular blogging systems, both can produce feeds, but WordPress does a much better job of it. Even using FeedBurner to improve the default Blogspot.com Atom feed doesn’t give you the granularity and control that WordPress does.
- Twitter is the world’s number one spot to break news or catch up the latest happenings and trends. Twitter has an API which you can use to search, but there are also third party tools that make searching and monitoring Twitter easier.
- Topsy is a “real-time search engine for the social web“. It can search Twitter and Google+, but doesn’t work with every social network. For Twitter it does a great job, better than the default Twitter search. It can create RSS feeds and it has an API.
- Tumblr gets a lot of press but I could list a number of reasons why I prefer WordPress and other publishing or cataloging tools. Tumblr’s RSS support isn’t as good as WordPress’s and it’s default search sucks. It is possible to use tools to monitor Tumblr for a particular topic or hashtag, such as this Yahoo Pipe, but it is definitely not as easy or as thorough as using Topsy to search Twitter.
- Pinterest was at first dismissed as a Tumblr clone but it has really caught on, especially with women. Text is a second class citizen on Pinterest with image being everything. Pinterest has built in support for RSS feeds which I was able to use as part of my miniature painting news aggregator. Pinterest works well for niches and markets that lend themselves to glossy product shots.
- Instagram is another billion dollar image sharing network. It is a less useful for following topics and trends but folks do use it to share photos, you can subscribe to an RSS feed for a hashtag, the problem with this approach is hashtags are not consistently and universally applied.
- Flickr has long had great support for RSS feeds and an API. Yet Pinterest and Instagram get all the press these days. Flickr is still popular with serious photographers. Yahoo sells them premium accounts. Pinterest is for sharing photos you found online, Flickr and Instagram for sharing photos you took yourself. Flickr’s RSS feeds use an enclosure which includes a thumbnail, this is something Pinterest should be doing too.
- Google Alerts are one of the oldest and still one of the best ways to monitor the Internet for a specific topic or keyword. I’ve used them for years, but every now and then an old dog can learn a new trick. It is possible to create a Google Alert and have it output RSS. Combined with say ‘site:domain.name‘ plus keywords and you can create custom search agents specifically for Tumblr or other sources.
- Feed of Feeds is a technique where you take lots of smaller independent information sources and merge them together into one easier to monitor feed. There are a number of blog networks in the miniature wargaming hobby, a few use tools like SimplePie to create one single feed that can be subscribed to, that includes the latest posts from a variety of sources. I built my own feed of feeds and it was how I discovered a probable bug in SimplePie.
- YouTube isn’t something I’ve traditionally tried to monitor, but it has an API which can produce an RSS feed. You can use these features to create a keyword search and then subscribe to the results, so if you want to see all the videos uploaded on a particular topic or in a particular category, no problem. Just like in Pinterest it is possible to subscribe to someone else’s uploads or favourites too. Videos get added and taken down quickly on YouTube and I don’t think the search result feed updates that frequently, but it is definitely another tool in the toolbox.
I looked at a lot of other tools and techniques, but if you have a killer one I missed, let me know in a comment. If you want to know more about how I used or created various feeds I’ll try to explain it better or find someone who can.
Update
Topsy now requires an API key to create the RSS feeds I talk about above and use in my news aggregator. These feeds were formerly a free service, but now they want you to transition to a commercial account. I have a 30 day free trial API key but unless they make an exception for me, I’m going to have to revise all my news aggregators and my other PHP code.
Related posts:
