• Print

Yahoo! Pipes: The Modules For Building Pipes

yahoo pipes
Yahoo! Pipes was released today with the goal of allowing people to easily mix, match, filter, sort and merge data sources into RSS feeds. These resulting RSS feeds are called Pipes and they allow you to do things like find all of the parks in your city or convert the news to Flickr photos. The product allows you to browse pipes, search for pipes, share pipes, or clone somebody else’s pipe. Tim wrote the launch piece, in this piece I am going to delve into the tools that they have made available to construct these pipes.

Pipes  are created in an editor on the website and are made up of modules. These modules are the tools that enable the mixing, sorting and merging of different data sources. The data sources are RSS feeds. Once created a Pipe is hosted at Yahoo! and can be accessed by any RSS reader via it’s unique URL.

pipes_edit_interface.jpg
The Pipes editor/creator is a really amazing piece of software. You are presented with a white, graphpaper-esque canvas to build your pipe on. A toolbox with a slew of potential modules is on the left, tabs for working with multiples pipes are across the top, and a debugger on the bottom. To build the Pipe you drag modules onto the canvas, enter the relevant data and connect them together via “wires”.

A good example Pipe is “Apartment Near Something“.  It allows you to input what you would like to be near (for example: “parks”), what city (for example: “Palo Alto, CA”), and how far (for example: 2 miles). It outputs GeoRSS of available apartments in a Palo Alto, CA that are near parks.

Click thru for a breakdown of the available modules that are used to create Pipes like this.

The modules for building Pipes are broken into these categories:

Data Sources: This is where you specify your RSS-formatted data inputs. Using the Fetch module you can pull in RSS or Atom feed that Yahoo! servers can access via HTTP. They have created wrappers for several services that did not expose their data as RSS. The included services are Yahoo! Local, Yahoo! Search, Flickr, and Google Base (complete with a field for your API key).

User Inputs: This where someone can enter input values before running a pipe. Y! Pipes provides five different types for easy structuring of data: Numbers, URL, Location, Text, and Date.

Operators: This is where the Unix of the system really comes out. :-) You have the following commands at your disposal:

  • For Each: Annotate – Annotates each item in the feed with the results of executing a nested module or pipe.
  • For Each: Replace – Replaces each item in the feed with the results of executing a nested module or pipe. (Example Pipe)
  • Count – Counts the number of items in the feed (Example Pipe)
  • Filter – Blocks or permits items that match the defined rules.
  • Truncate – Limits the number of items that passes through it.
  • Sort – You can sort items in a feed.
  • Babelfish – Translates to and from select languages
  • Location Extractor – Detects location entities in text and geocodes them. This is using technology from their WhereOnEarth acquisition.
  • Union – Merges two feeds
  • Unique – Filters out duplicates

URL – This category only contains the URL Builder Module. With this module you can construct custom urls that pass along data from the User Input module or other module. For example, the URL http://sfbay.craigslist.org/search/aap?query=Palo+Alto&format=rss is constructed by this URL Builder Module and this Location User Input Module.
pipes_urlblder.jpg
Palo Alto is the default, but when running this Pipe you could put in any city you wished. To fetch a feed with this module you must connect it to a Fetch Module.

String: This category only contains the module String Concatenate which joins multiple item attributes into a single string.

Date:
This category only contains the module Date Formatter which formats a date-time value piped in from another module as a string.

My Pipes:
This is where we see some Unix resurfacing in the design. Your own pipes and other people’s pipes can be called as data sources or as functions in your own pipe. Very slick and powerful.

In my next post I am going to deconstruct the Near My Apartment Pipe.

Upate: Yahoo! Pipes now has a listing of all the modules in the help section; it wasn’t up when I was writing this.

This is post is the third in a series on the Pipes product. Here are the other posts on Radar:

tags:
  • http://kevnull.com Kevin Cheng

    Thanks for the write-up, Brady. As you might guess, the module categories that only have one item in them (specifically, URL, String and Date) are organized that was because we already have a few other related operators in mind for each of those categories. Of course, if you find yourself trying to build a Pipe and in need of some kind of module, we’d love to put together a wishlist.

  • Romain

    I think someone at Yahoo! has used Apple’s Quartz Composer because the UI looks pretty familiar…;-)

  • http://captsolo.net/info/ Uldis Bojars

    Do you happen to know what data format are Yahoo! Pipes using to transfer information within a pipeline?

    If that is RSS (+ some extensions) it may be limiting use of Pipes for richer data sources which would loose information if converted down to RSS/Atom.

    A very nice user interface paradigm though.

  • http://mrspeaker.webeisteddfod.com/ Mr Speaker

    I was playing with Pipes, and something went pear-shaped and a bunch of JSON script took over my screen. I assume that’s what they are piping around.

  • http://usefulvideo.blogspot.com/ usefulvideo

    Good overview. In my experience, the URL Builder can be quite tricky.

  • http://gruks.com/ Karry

    After reading Tim;s post i started building my first pipe. your this post also helped me alot thanks

  • http://dasht-exp-1a.com Thomas Lord

    “Pipes” alone are not unix. With this kind of pipe, basically you can connect up a pipeline yourself — but only other people can install the programs you are piping together. This is “browser as shell” not really a programmable web. It still insists on a high priesthood, so to speak, in the back office.

    -t