Yahoo! Pipes: Some Tips

Yahoo! has released a new RSS remixer named Pipes. Right now it’s got some performance issues, but once those are worked out you should go play with the service.
These are some facts and tips about Pipes that I learned while talking with the developers, playing with the app as they were making changes and from reading the documentation.

  • To change the name of a Pipe, open it in edit mode and click on the tab. It will become a text field that you can modify and save.
  • Pipes will output GeoRSS. They plan to release a map viewer soon. In the meantime you can consume GeoRSS over at Mapufacture.
  • You can use Dapper (or Dappit) and OpenKapow to scrape sites. They both output RSS feeds that can be consumed by Pipes.
  • They have a slick debugger that allows you to step through your modules one at a time and view the data at each step.
  • The following browsers are suported: Firefox, Webkit, and IE6. Known Problems: Pipes is known not to work in Opera. Safari has minor issues. IE6 currently displays problems with the For Each module. IE7 has not been extensively tested.
  • In addition to RSS, Pipes also output in JSON.
  • The Pipes crawler respects the Robot Exclusion Standard. It has the User-agent: Pipes-Crawler Yahoo Pipes.

This is post is the fourth in a series on the Pipes product. Here are the other posts on Radar:


Update –
Daniel Raffel of the Yahoo! TechDev Team, the creators of Pipes, wrote to me:

I just saw your latest post on radar about pipes and wanted to point out
that the correct user agent to block is “Yahoo Pipes” NOT “Pipes-Crawler”.

Even more details on that are listed here:
http://pipes.yahoo.com/docs/troubleshooting

Another cool feature I thought I’d mention is that you can search for feeds
in the feed finder in the top right corner, I’ve attached a pic. You can
drag the results to the canvas.

pipes_feed.finder.jpg

Finally, we have some “easter eggs” for folks to discover but they are
hidden as usual, and, well, we can’t spoil the fun.

tags:
  • http://ww.jperla.com/ Joseph Perla

    Brady,
    I think by the Dapper link you mean to use the url dappit.com rather than dapper.com.
    Cheers,
    j

  • http://www.1060.org/blogxter/publish/5 Steve Loughran

    Having played with it yesterday, I think it would be nice if feeds could come with authentication. I couldnt get my gmail atom feed into a pipe, so I couldnt get an RSS feed of my email translated into french.

    with authentication, you could then make many more private atom/rss feeds that are then chained together in interesting ways. Admittedly, I dont know what this does for the pipes business model, but it could be cute nonetheless.

  • http://www.malaiac.net Malaiac

    It does work in Opera, but there are some glitches (moving the boxes, connecting them, etc.)

    Great product, anyway, in two minutes yesterday, I did a simple filtered Y! search with parameters, on two keywords.

  • http://www.malaiac.net Malaiac

    You may have made a mistake for dapper :
    it’s http://www.dappit.com/ and not -www.dapper.com/ (fashion clothes) !

  • Jon

    Dapper should be dapper.net or dappit.com.

  • http://krishami.blogspot.com Krishna Kumar

    I was able to create a pipe in IE 7.0, but the description of the modules were not displaying properly.

    I also tested with Firefox 2.0 and it was really displaying very poorly.

  • http://search-search-and-searchme.blogspot.com/ Harish

    I really think the best way to use this new product is to have an entire community of users interacting with each other and creating content.

    I have tried to kick start something of the sort. Guess we will just have to wait and see where it goes.

  • http://usefulvideo.blogspot.com/ usefulvideo

    These were really useful tips. I like using feedity.com to scrape sites because it doesn’t require you to log in. I created several video tutorials for Yahoo Pipes. With your permission, here is the link:

    http://usefulvideo.blogspot.com/2007/02/yahoo-pipes-tutorials.html

  • mosaic blue

    I really have trouble understanding how the thing runs. I tried using a pipe yesterday and I just couldn’t get the damn thing to do anything. I hope someone will give me a word how the thing works. I have not ever ever touched any UNIX environment at all. That said, I would greatly greatly appreciate if somebody would give me the time of day as to how this thing runs. I want “in” on the excitement. I hope someone tags me up and helps me. Thank you and hope everyone has a very good day.

  • http://www.apatar.com/ JR

    Apatar Open Source can be used to compliment Yahoo Pipes. It can extract data from on-premise systems and feed mashups in situations where large amounts of data need to be extracted from databases like MySQL or Oracle, transformed and loaded into a target like Amazon or SalesForce.com, or into an XML feed.

  • morganusvitus

    The site looks great ! Thanks for all your help ( past, present and future !)

  • http://www.web-articles.info atila

    I wanted to combine all favorite news feeds but Yahoo pipes not working in I.E. 7, but in I.E. 6, worked fine.

  • http://www.internetmarketingcircle.com Dennis

    One of the main things that Yahoo Pipes has done is it’s made a difficult programming chore more accessible to users. So many programming languages make assembly language more accessible. But, there is still quite a learning curve for the newbie user :)

    Imagine being able to take any feed, performing operations on them and outputting another feed.

  • http://www.maxdirectory.eu ibolya

    Wow. This is the *perfect* tool for me to illustrate to my coworkers a number of concepts relating to blogging, rss, and the universal database of participation all of us have been building.

    Sure you could all of what Pipes provided with some Python or Perl fairly quickly – if you were a scripter like myself.

    But this opens it up to almost anyone. And that makes it huge.

    On a personal note, this is precisely the kind of tool that will enable me to provide useful views into the conversation talking place in the Philly Future regional blog community.

    I don’t see an OPML input there, but I’ve just dipped my toes in this morning.

    Great work Yahoo! :)

  • http://www.lost-series.com LS-SAM

    wow – great little application, makes life a little bit easier. Every little helps? hehe

  • http://www.dminternational.biz DirMax Web Directory

    Nice tips that will come handy. Thanks!

  • http://www.mobuniverse.com mobile softwares

    hi Brady Forrest, Very nice Tips. I really reserching on yahoo pipes. Your information is very useful . I think this service is going to be a hit soon

  • http://www.mobuniverse.com mobile softwares

    hi Brady Forrest, Very nice Tips. I really reserching on yahoo pipes. Your information is very useful . I think this service is going to be a hit soon

  • http://afghan-artist.com/Desktop-Tower-Defense-Strategies/ MaxAzi

    I read something very same somwhere else in oreilly article can someone please tell me what is the difference between that article and this one over here? arent they basically the same… and could someone please put up detailed version of Pipes as many people “laymen” specially donno what exactly pipes are lol not to confuse with other pipe in household..

  • http://www.e-articles.info Andy

    Reading the tips Brady shared with us, I considered necessary to share a resource containing short video tutorials on how to work with pipes.

    usefulvideo.blogspot.com/2007/02/yahoo-pipes-tutorials.html

    youtube.com/watch?v=2pVxqgt5jTs

  • http://www.youtubebestvideos.com Tube Man

    Thank you Brady for useful tips about Yahoo’s Pipes service. It helps me a lot. I think the Yahoo Pipes servie is a really great project!

  • http://www.gohomoney.com Goody

    Nice tips there, Brady. The other Oreilly article article on Yahoo Pipes explains the concept but you make it even clearer through these tips.

  • http://www.tupac-amaru.com Tupac

    Great collection of Tipac there mate, thankyou!

  • http://www.proxidude.info Tom

    Apatar Open Source can be used to compliment Yahoo Pipes. It can extract data from on-premise systems and feed mashups in situations where large amounts of data need to be extracted from databases like MySQL or Oracle, transformed and loaded into a target like Amazon or SalesForce.com, or into an XML feed.

  • http://www.co2calculateur.com mathew hades

    can you please tell me what are the performance issues with pipes. are the issues still there?

  • http://www.lijo.in Lijo

    what is the difference between GeoRSS and simple RSS. is geo RSS a type of feed?

  • http://radar.oreilly.com/brady brady

    GeoRSS is an extension on RSS. Like MediaRSS. Check out georss.org for more info.

  • http://www.dirbuzz.com zeb

    Nice tips Brady. Here is a video tutorial that gave me a good start to work on pipes : http://www.jumpcut.com/fullscreen?id=594F555C568011DC9D24000423CEF5B0&type=movie

  • http://www.funnyyoutubevideo.com Tube

    Thank you for the tips, Brady. But I’m curious about Yahoo Pipes service working with Opera. Is it already possible for an Opera user to use it?

  • http://www.makequick.com Nick

    Nice video tutorial Zeb, thanks for the link!

  • http://www.jrconsumer.com Motorhomes

    what is the difference between GeoRSS and simple RSS. is geo RSS a type of feed?

    Thanks,

    RV

  • http://www.corecreativegroup.com The Muse

    The Pipe is already a big hit in my country. Seems the concept spread like wildfire. Could this be the site that lit that fire?

  • http://www.bodfeld-apotheke.de Apotheke

    Thanks for very interesting article. btw. I really enjoyed reading all of your posts. It’s interesting to read ideas, and observations from someone else’s point of view… makes you think more. It’s really good written and I fully agree with You on main issue, btw. I must say that I really enjoyed reading all of Your posts. It’s interesting to read ideas, and observations from someone else’s point of view…

  • http://www.dreaddisease.biz Horatiu Vasile

    Hi, I¬¥m looking for a solution to mix different feeds by date using different date formats or even without date coded in the feed…
    Does anyone know how to solve this?

  • http://www.jrconsumer.com Motorhome

    Thank you Brady for useful tips about Yahoo’s Pipes service. It helps me a lot. I think the Yahoo Pipes servie is a really great project! Again, very cool post!

  • http://whatsonnews.com whatsonnews

    Really great product from the yahoo stable but i’ve got issues with the filtering side. U’re not able to prevent the same news item displaying twice or more when more feeds are merged to produce one pipe.

    Hope they rectify this.

  • dmeanea

    @whatsonnews: use the “unique” operator

  • http://www.ytamil.com Ytamil.com

    nice info, i am learning about yahoo’s pipes service