Yahoo! Pipes: The Modules For Building Pipes

Yahoo! Pipes was released today with the goal of allowing people to easily mix, match, filter, sort and merge data sources into RSS feeds. These resulting RSS feeds are called Pipes and they allow you to do things like find all of the parks in your city or convert the news to Flickr photos. The product allows you to browse pipes, search for pipes, share pipes, or clone somebody else's pipe. Tim wrote the launch piece, in this piece I am going to delve into the tools that they have made available to construct these pipes.

Pipes  are created in an editor on the website and are made up of modules. These modules are the tools that enable the mixing, sorting and merging of different data sources. The data sources are RSS feeds. Once created a Pipe is hosted at Yahoo! and can be accessed by any RSS reader via it's unique URL.

The Pipes editor/creator is a really amazing piece of software. You are presented with a white, graphpaper-esque canvas to build your pipe on. A toolbox with a slew of potential modules is on the left, tabs for working with multiples pipes are across the top, and a debugger on the bottom. To build the Pipe you drag modules onto the canvas, enter the relevant data and connect them together via "wires".

A good example Pipe is "Apartment Near Something".  It allows you to input what you would like to be near (for example: "parks"), what city (for example: "Palo Alto, CA"), and how far (for example: 2 miles). It outputs GeoRSS of available apartments in a Palo Alto, CA that are near parks.

Click thru for a breakdown of the available modules that are used to create Pipes like this.

The modules for building Pipes are broken into these categories:

Data Sources: This is where you specify your RSS-formatted data inputs. Using the Fetch module you can pull in RSS or Atom feed that Yahoo! servers can access via HTTP. They have created wrappers for several services that did not expose their data as RSS. The included services are Yahoo! Local, Yahoo! Search, Flickr, and Google Base (complete with a field for your API key).

User Inputs: This where someone can enter input values before running a pipe. Y! Pipes provides five different types for easy structuring of data: Numbers, URL, Location, Text, and Date.

Operators: This is where the Unix of the system really comes out. :-) You have the following commands at your disposal:

• For Each: Annotate - Annotates each item in the feed with the results of executing a nested module or pipe.
• For Each: Replace - Replaces each item in the feed with the results of executing a nested module or pipe. (Example Pipe)
• Count - Counts the number of items in the feed (Example Pipe)
• Filter - Blocks or permits items that match the defined rules.
• Truncate - Limits the number of items that passes through it.
• Sort - You can sort items in a feed.
• Babelfish - Translates to and from select languages
• Location Extractor - Detects location entities in text and geocodes them. This is using technology from their WhereOnEarth acquisition.
• Union - Merges two feeds
• Unique - Filters out duplicates

URL - This category only contains the URL Builder Module. With this module you can construct custom urls that pass along data from the User Input module or other module. For example, the URL http://sfbay.craigslist.org/search/aap?query=Palo+Alto&format=rss is constructed by this URL Builder Module and this Location User Input Module.

Palo Alto is the default, but when running this Pipe you could put in any city you wished. To fetch a feed with this module you must connect it to a Fetch Module.

String: This category only contains the module String Concatenate which joins multiple item attributes into a single string.

Date: This category only contains the module Date Formatter which formats a date-time value piped in from another module as a string.

My Pipes: This is where we see some Unix resurfacing in the design. Your own pipes and other people's pipes can be called as data sources or as functions in your own pipe. Very slick and powerful.

In my next post I am going to deconstruct the Near My Apartment Pipe.

Upate: Yahoo! Pipes now has a listing of all the modules in the help section; it wasn't up when I was writing this.

This is post is the third in a series on the Pipes product. Here are the other posts on Radar:

• Pipes and Filters for the Internet
• Yahoo! Pipes: Some Tips
• Yahoo! Pipes: Deconstructing a Pipe