Yahoo! Pipes: Deconstructing a Pipe
Yahoo! has just released a site for hosting and mixing RSS feeds; read Tim's launch post for details. In my previous post Yahoo! Pipes: The Modules For Building Pipes I explained the tools available for building a pipe and mentioned the Apartment Near Something pipe. It allows you to input what you would like to be near (for example: "parks"), what city (for example: "Palo Alto, CA"), and how far (for example: 2 miles). It outputs GeoRSS of available apartments in a Palo Alto, CA that are near parks. In this post I am going to step through this pipe. If you want to follow along you can check the debugger in the editor to see the output at each step.
- You are prompted to fill in 3 fields: your location (palo alto), what you would like to be near (parks), and how far away they can be (2 miles).
- The Location User Input module takes the location field and passes it to the URL Builder. It constructs the following URL: http://sfbay.craigslist.org/search/aap?query=Palo+Alto&format=rss
- The URL is passed to the Fetch Module.
- The resulting RSS feed is passed to the Location Extractor Module. The entries that have text relating to location get geotagged.
- The updated feed get passed to a Filter Module. Only locations that have been extracted with a confidence of 80 or higher can pass through.
- The filtered feed is passed to a For Each: Annotate Module; it has a nested Yahoo! Local Search Data Source Module. The Local module accepts "parks" as an input from the Text User Input Module.
- Then the each item of the feed is checked to see how close it is to area parks.
- The updated feed is passed through another Filter Module. This Filter Module takes input from a Number User Input Module to filter out rental properties that aren't within 2 miles of a park.
- The Pipe is complete and can be consumed as GeoRSS
I want to thank Pasha Sadri, the principal software engineer of Pipes, who answered my questions and explained some concepts along the way.
tags: web 2.0
| comments: 33
| Sphere It
submit: 
0 TrackBacks
TrackBack URL for this entry: http://blogs.oreilly.com/cgi-bin/mt/mt-t.cgi/5223
Comments: 33
Very nice. The site must be getting hammered as a result of the attention, though, because it isn't loading.
Is the site actually up right now? Could you comment on the amount of traffic Pipes is getting today? (this question is going to the Pipes guys, sorry to leave it here...)
John,
The site looks to be live, the problem is in getting to it. I've only been able to get the page to load a couple of times. All other attempts time out.
I got the site to load moments ago, and was immediately underwhelmed.
This is beta software, so a bit of this is to be expected, but...
- Click 'Create Pipe' in fact creates 'System Error'
- The expandable tree in the left frame doesn't (expand)
- 'Browse for Pipes' generates a 'Sorry, we can't find a Pipe with that name' error before I submitted any search criteria. In fact, I never saw anything except that error; no search box, no tree to browser, nothing.
If this is simply an experiment by some folks inside Yahoo, then I would assess this as a partial success.
Plenty of interest was generated, but the project clearly had insufficient back-end resources to handle an O'Reilly and Techcrunch mention on the same day. Too bad all this initial interest is being met with such a poorly performing application.
First impressions are incredibly difficult to change.
Looks very good. I made my first pipe yesterday after playing around with it for some time. I was confused at first but after looking at some other pipes that had been made it all came together. I think there is going to be great potential with the Yahoo Pipes after even more module's will be added.
well, all of this is very nice, but the only raeal winner here is osman bin laden. sure, its good to ack the unix pipe folks, but what is the pint? ecvry search feed out there lets you "customize' it for your self.
Very useful breakdown. It would be nice of the location extraction module were more transparent, so you could know exactly what it was doing.
Looks good to me. I made my first pipe two days ago after playing around with it for some time. I have hound very nice tutuorial on youtube. After adding more macules it will be very cool stuff.
"The updated feed get passed to a Filter Module. Only locations that have been extracted with a confidence of 80 or higher can pass through."
This value can be chanced?
Anyone have an example of turning a phpBB (or any) forum into an RSS feed? I've got a favorite forum that's not available via RSS/Atom/etc. It looks as though Pipes might make it accessible for me but (at least at first glance) I can't figure out how!
ibolya, the forum must have implemented the rss technology in order to be able to use it with the new Yahoo pipes. You may use a free online rss generator tool like
wotzwot.com/rssxl.php
in order to generate the rss pages, then use the yahoo pipes.
I also found a comprehensive tutorial on how to create quality content using the Yahoo pipes
video.yahoo.com/video/play?vid=289820
The video is by Sadri and Ed Ho from the Yahoo! Developer Network. It shows how to create a Pipe and there are some examples too.
After reading yours and Tim's post on Yahoo Pipes . I decided to make my Pipe a week ago. Really loved it although it was a little difficult in starting but that was really awesome. To generate rss feed i used some online program which was much better.
I'm interested in a better program too. I'm experiencing a lot of problems with what I am using so far.
this sounds interesting, i've been looking for something like this for a mobile internet site i'm doing at the moment.
thanks again
Looks very good. I made my first pipe yesterday after playing around with it for some time. I was confused at first but after looking at some other pipes that had been made it all came together. I think there is going to be great potential with the Yahoo Pipes after even more module's will be added.
I cant it get working :( Its look very nice but I can t made one which will be looking a little as other pipes
Greg
I built a few months ago a similar service. Yahoo! is ruining me :-D Seriously, Google and Y! are covering broader and broader spaces of the web, and small companies are being destroyed along the way. Just an observation, not a blow at them.
Looks very interesting.
Has anyone an idea how to put a site into RSS-Feed which isn't offering a Feed by itself?
Pipes is even more powerful now when has the reg exp operator. I'm reading this tutorial ( http://theytookmystapler.blogspot.com/2007/09/yahoo-pipes-case-study-of-using-regex.html ) on it, I will use Pipes with the reg exp operator to change the titles of my Flickr pictures.
It is new for me (and not only for me). Every Module in the Pipes Editor now has a enhanced description including details about what it does, the types of data it can input and output and links to examples. It is new for me (and not only for me I hope).
It took some time to get to know yahoo pipe, but after that, I was very positive surprised.
Hope for further developements.
And I want to say that using regular expressions is very important in Yahoo Pipes. This thing give wide abilities to organize content
woooohooo !! I've been piping (mind the linux punt :)) for an hour now, it just gets more and more enjoyable.
Cheers Yahoo .
"PS: the value of 80 can not be changed to higher."
true, I had the same problem
I tried to use Pipes, but I am not strong in regular expressions, that is why for me more comfortable to use the own software, written specially for me.
Post A Comment:
STAY CONNECTED
RECENT COMMENTS
- Smystery on Yahoo! Pipes: Deconstructing a Pipe: I just know about Yahoo...
- Mps on Yahoo! Pipes: Deconstructing a Pipe: thanks, this is very ni...
- David V on Yahoo! Pipes: Deconstructing a Pipe: I tried to use Pipes, b...
- Donald on Yahoo! Pipes: Deconstructing a Pipe: Looks very interesting....
- Simone Kurzreisen on Yahoo! Pipes: Deconstructing a Pipe: "PS: the value of 80 ca...
- איחסון אתרים on Yahoo! Pipes: Deconstructing a Pipe: woooohooo !! I've been ...
- Random on Yahoo! Pipes: Deconstructing a Pipe: And I want to say that ...
- Random on Yahoo! Pipes: Deconstructing a Pipe: Thanks, interesting thi...
- JoM. on Yahoo! Pipes: Deconstructing a Pipe: It took some time to ge...
- Daniel on Yahoo! Pipes: Deconstructing a Pipe: It is new for me (and n...
RADAR TOPICS
Pasha Sadri [02.08.07 01:19 AM]
Thanks for the post Brady.
We hope to expand the scope of pipes by adding more modules and support for different data sources in the near future. If you or your readers have suggestions or requests, please post them on our feedback forums:
http://suggestions.yahoo.com/?prop=Pipes
Pasha (from the Pipes team)