Yahoo! Pipes: Deconstructing a Pipe

pipes_edit.jpg
Yahoo! has just released a site for hosting and mixing RSS feeds; read Tim’s launch post for details. In my previous post Yahoo! Pipes: The Modules For Building Pipes I explained the tools available for building a pipe and mentioned the Apartment Near Something pipe. It allows you to input what you would like to be near (for example: “parks”), what city (for example: “Palo Alto, CA”), and how far (for example: 2 miles). It outputs GeoRSS of available apartments in a Palo Alto, CA that are near parks. In this post I am going to step through this pipe. If you want to follow along you can check the debugger in the editor to see the output at each step.

  1. You are prompted to fill in 3 fields: your location (palo alto), what you would like to be near (parks), and how far away they can be (2 miles).
  2. The Location User Input module takes the location field and passes it to the URL Builder. It constructs the following URL: http://sfbay.craigslist.org/search/aap?query=Palo+Alto&format=rss
  3. The URL is passed to the Fetch Module.
  4. The resulting RSS feed is passed to the Location Extractor Module. The entries that have text relating to location get geotagged.
  5. The updated feed get passed to a Filter Module. Only locations that have been extracted with a confidence of 80 or higher can pass through.
  6. The filtered feed is passed to a For Each: Annotate Module; it has a nested Yahoo! Local Search Data Source Module. The Local module accepts “parks” as an input from the Text User Input Module.
  7. Then the each item of the feed is checked to see how close it is to area parks.
  8. The updated feed is passed through another Filter Module. This Filter Module takes input from a Number User Input Module to filter out rental properties that aren’t within 2 miles of a park.
  9. The Pipe is complete and can be consumed as GeoRSS

This was a relatively simple Pipe that was created by the Pipes team. It works well as an example because it pulls in most of the common features.
I want to thank Pasha Sadri, the principal software engineer of Pipes, who answered my questions and explained some concepts along the way.

tags:
  • http://pashasadri.wordpress.com/ Pasha Sadri

    Thanks for the post Brady.

    We hope to expand the scope of pipes by adding more modules and support for different data sources in the near future. If you or your readers have suggestions or requests, please post them on our feedback forums:

    http://suggestions.yahoo.com/?prop=Pipes

    Pasha (from the Pipes team)

  • http://freepository.com John Minnihan

    Very nice. The site must be getting hammered as a result of the attention, though, because it isn’t loading.

    Is the site actually up right now? Could you comment on the amount of traffic Pipes is getting today? (this question is going to the Pipes guys, sorry to leave it here…)

  • http://stickiwidgets.com Ming

    John,

    The site looks to be live, the problem is in getting to it. I’ve only been able to get the page to load a couple of times. All other attempts time out.

  • http://freepository.com John Minnihan

    I got the site to load moments ago, and was immediately underwhelmed.

    This is beta software, so a bit of this is to be expected, but…

    – Click ‘Create Pipe’ in fact creates ‘System Error’

    – The expandable tree in the left frame doesn’t (expand)

    – ‘Browse for Pipes’ generates a ‘Sorry, we can’t find a Pipe with that name’ error before I submitted any search criteria. In fact, I never saw anything except that error; no search box, no tree to browser, nothing.

    If this is simply an experiment by some folks inside Yahoo, then I would assess this as a partial success.

    Plenty of interest was generated, but the project clearly had insufficient back-end resources to handle an O’Reilly and Techcrunch mention on the same day. Too bad all this initial interest is being met with such a poorly performing application.

    First impressions are incredibly difficult to change.

  • http://breakingnews1.blogspot.com Breaking News1

    Looks very good. I made my first pipe yesterday after playing around with it for some time. I was confused at first but after looking at some other pipes that had been made it all came together. I think there is going to be great potential with the Yahoo Pipes after even more module’s will be added.

  • maria a

    well, all of this is very nice, but the only raeal winner here is osman bin laden. sure, its good to ack the unix pipe folks, but what is the pint? ecvry search feed out there lets you “customize’ it for your self.

  • http://usefulvideo.blogspot.com/ usefulvideo

    Very useful breakdown. It would be nice of the location extraction module were more transparent, so you could know exactly what it was doing.

  • http://www.t4tw.info Sebastian Snopek

    Looks good to me. I made my first pipe two days ago after playing around with it for some time. I have hound very nice tutuorial on youtube. After adding more macules it will be very cool stuff.

  • http://www.maxwendesign.eu Michael

    “The updated feed get passed to a Filter Module. Only locations that have been extracted with a confidence of 80 or higher can pass through.”

    This value can be chanced?

  • http://www.maxwebdesign.eu Mike

    PS: the value of 80 can not be changed to higher.

  • http://www.maxdirectory.eu ibolya

    Anyone have an example of turning a phpBB (or any) forum into an RSS feed? I’ve got a favorite forum that’s not available via RSS/Atom/etc. It looks as though Pipes might make it accessible for me but (at least at first glance) I can’t figure out how!

  • http://www.e-articles.info Andy Dinu

    ibolya, the forum must have implemented the rss technology in order to be able to use it with the new Yahoo pipes. You may use a free online rss generator tool like
    wotzwot.com/rssxl.php
    in order to generate the rss pages, then use the yahoo pipes.
    I also found a comprehensive tutorial on how to create quality content using the Yahoo pipes
    video.yahoo.com/video/play?vid=289820
    The video is by Sadri and Ed Ho from the Yahoo! Developer Network. It shows how to create a Pipe and there are some examples too.

  • http://www.uparticles.com James

    After reading yours and Tim’s post on Yahoo Pipes . I decided to make my Pipe a week ago. Really loved it although it was a little difficult in starting but that was really awesome. To generate rss feed i used some online program which was much better.

  • http://radar.oreilly.com/brady brady

    @james, which one?

  • http://www.mygreencorner.com/freewordpressthemes.php Dugu

    I’m interested in a better program too. I’m experiencing a lot of problems with what I am using so far.

  • http://livecricketscore.mobi ukcricketfan

    this sounds interesting, i’ve been looking for something like this for a mobile internet site i’m doing at the moment.

    thanks again

  • http://www.tupac-amaru.com Tupac

    Interesting read, I like it.

  • http://www.bloomie.info Luke

    Looks very good. I made my first pipe yesterday after playing around with it for some time. I was confused at first but after looking at some other pipes that had been made it all came together. I think there is going to be great potential with the Yahoo Pipes after even more module’s will be added.

  • http://www.avsh.pl Greg

    I cant it get working :( Its look very nice but I can t made one which will be looking a little as other pipes

    Greg

  • http://www.poemas-del-alma.com Julian

    I built a few months ago a similar service. Yahoo! is ruining me :-D Seriously, Google and Y! are covering broader and broader spaces of the web, and small companies are being destroyed along the way. Just an observation, not a blow at them.

  • http://www.gegen-langeweile-im-internet.de/ Tony

    Looks very interesting.
    Has anyone an idea how to put a site into RSS-Feed which isn’t offering a Feed by itself?

  • Bo Aphran

    Pipes is even more powerful now when has the reg exp operator. I’m reading this tutorial ( http://theytookmystapler.blogspot.com/2007/09/yahoo-pipes-case-study-of-using-regex.html ) on it, I will use Pipes with the reg exp operator to change the titles of my Flickr pictures.

  • http://blog.davidjablonsky.com David

    This is good idea create this page.

  • http://www.mifdesign.com/ Daniel

    It is new for me (and not only for me). Every Module in the Pipes Editor now has a enhanced description including details about what it does, the types of data it can input and output and links to examples. It is new for me (and not only for me I hope).

  • http://www.jomik.de/ JoM.

    It took some time to get to know yahoo pipe, but after that, I was very positive surprised.
    Hope for further developements.

  • http://noulake.com/blog Random

    Thanks, interesting thing! I think I will use this service a lot

  • http://mp3audioz.com Random

    And I want to say that using regular expressions is very important in Yahoo Pipes. This thing give wide abilities to organize content

  • http://www.◊ê◊ô◊ó◊°◊ï◊ü.net ◊ê◊ô◊ó◊°◊ï◊ü ◊ê◊™◊®◊ô◊ù

    woooohooo !! I’ve been piping (mind the linux punt :)) for an hour now, it just gets more and more enjoyable.

    Cheers Yahoo .

  • http://www.reisen-to-go.de/ Simone Kurzreisen

    “PS: the value of 80 can not be changed to higher.”
    true, I had the same problem

  • http://mp3recs.com/ Donald

    Looks very interesting.
    Good service yahoo creation.

  • http://musicdigits.com/ David V

    I tried to use Pipes, but I am not strong in regular expressions, that is why for me more comfortable to use the own software, written specially for me.

  • http://mp3sforum.com Mps

    thanks, this is very nice tutorial

  • http://smystery.my24hours.net/ Smystery

    I just know about Yahoo! Pipes, sound interesting an fun. Thanks