Four short links: 5 February 2010

Public Domain, Science Code, Bad Crypto, Javascript Grids

1. The Public Domain Manifesto -- eloquent argument in favour of the public domain. (via BoingBoing)
2. Clear Climate Code -- project to write and maintain software for climate science, with an emphasis on clarity and correctness. What a wonderful way for coders who aren't scientists to contribute to open and better science. (via the interesting OKFN blog)
3. Don't Hash Secrets -- One area of secure protocol development that seems to consistently yield poor design choices is the use of hash functions. What I’m going to say is not 100% correct, but it is on the conservative side of correct, so if you follow the rule, you (probably) can’t go wrong. You might be considered overly paranoid, but as they say, just because you’re paranoid doesn’t mean they’re not after you. So here it is: Don’t hash secrets. Never. No, sorry, I know you think your case is special but it’s not. No. Stop it. Just don’t do it. You’re making the cryptographers cry.
4. Javascript Grid Editors -- nice wrapup of available Javascript editable grid components, divided into "data driven", "light edit", and "spreadsheet". (via joshua on Delicious)

One hundred eighty degrees of freedom: signs of how open platforms are spreading

I was talking recently with Bob Frankston, who has a distinguished history in computing that goes back to work on Multics, VisiCalc, and Lotus Notes. We were discussing some of the dreams of the Internet visionaries, such as total decentralization (no mobile-system walls, no DNS) and bandwidth too cheap to meter. While these seem impossibly far off, I realized that computing and networking have come a long way already, making things normal that not too far in the past would have seemed utopian.

(continue reading)

Four short links: 4 February 2010

Personal Ad Preferences, Android Kernel, EC2 Deconstructed, Symbian Opened

1. Google Ad Preferences -- my defaults look reasonable and tailored to my interest. Creepy but kinda cool: I guess that if I have to have ads, they should be ones I'm not going to hate. (via rabble on Twitter)
2. Android and the Linux Kernel -- the Android kernel is forked from the standard Linux kernel, and a Linux kernel maintainer says that Google has made no efforts to integrate. (via Slashdot)
3. On Amazon EC2's Underlying Architecture -- fascinating deconstruction of the EC2 physical and virtual servers, without resorting to breaking NDAs. (via Hacker News)
4. First Full Open Source Symbian Release (BBC) -- source code will be available for download from the Symbian Foundation web site as of 1400GMT. Nokia bought Symbian for US$410M in 2008 (for comparison, AOL bought Netscape for $4.2B in 1999 but the source code tarball had been escape-podded from the company a year before the deal closed). This makes Symbian more open than Android, says the head of the foundation: "About a third of the Android code base is open and nothing more,” says Williams. “And what is open is a collection of middleware. Everything else is closed or proprietary.” (quote from Wired's story).

What Facebook's HipHop means for developers and businesses

O'Reilly PHP author Kevin Tatroe puts Facebook's PHP project in context

Facebook's PHP overhaul, HipHop, reportedly cut CPU usage on the company's servers by around 50 percent. You don't have to be a programmer to understand that kind of result.

Facebook says it wants to share that efficiency -- and presumably influence a few things along the way -- so it's setting HipHop loose as an open source project. I asked Kevin Tatroe, co-author of O'Reilly's Programming PHP, to weigh in on HipHop's functionality and its broader applications.

Mac Slocum: How will HipHop help programmers?

Kevin Tatroe: One of PHP's greatest strengths is its expansive leniency. But for very large code bases, it can also be somewhat problematic. For example, while you can change the type of data stored in a variable mid-script, I'd wager that the vast majority of the time, it's a mistake.

Those are the kinds of things HipHop's analysis and type inference steps will find. For that reason, I can even see running sites through HipHop that don't need footprint savings, just as a sanity check.

MS: How about businesses that rely on PHP. Does HipHop offer them any utility?

KT: Certainly. At its best, PHP scales very well by running on teeming hordes of cheap servers. As great as that strategy has turned out to be, running on half as many teeming hordes of cheap servers has clear benefits in deployment costs and costs to maintain.

There's also benefits at the other end of the scale. I've seen more companies than I can count stick with one deployment server when they really ought to be scaling up to two or three. There's a "leap of faith" barrier there. Companies say: "Obviously, we can't deploy on zero servers, so one seems fine. But two? We're just a small operation. Can't you make your code work better?"

MS: Facebook calls HipHop a "source code transformer." In plain terms, what is that?

KT: It takes the PHP code written by PHP programmers and converts it to C++ code, which is then compiled by g++ into machine code.

In doing so, it has to disallow certain PHP language features, like eval(). And it runs a pass to determine what type each variable in your PHP code should be in C++.

PHP does not require you to state up front what kind of data you're going to store in a variable. It lets you change the type of data stored in a given variable willy-nilly, which is not necessarily the best idea.

MS: Facebook says HipHop reduced CPU load on its servers by about 50 percent. Any idea how it does that?

KT: Running native, compiled C++ takes less processing effort to run than PHP's scripts via an opcode virtual machine (such as using Zend Engine). That's because it's skipping the virtual machine entirely.

It's telling that Facebook didn't mention any HipHop speed increases. It'd be surprising if there weren't any measurable speed improvements, but their primary focus seems to be: use less cheap hardware to run the same site.

MS: How easy -- or hard -- do you think it will be for other companies to take advantage of HipHop?

KT: This all depends on the tools. Certainly, any organization with deployments large enough to really notice much out of this will have devs comfortable with the more traditional "write, compile, test, deploy" cycle.

PHP also attracts a large percentage of folks who've never had to compile anything in their lives. But these same people aren't necessarily afraid of the command line. If the tools are simple enough that people aren't scared off, it should be pretty simple to get a site up and running using HipHop.

Note: Kevin's comments were condensed and edited from a longer interview.

Can open source guide a moon mission?

The Open Luna Foundation has a $500 million plan to build a moon station.

I'm a sucker for audacious ideas. Big, huge things with a hint of insanity. And if you put those ideas in space I get really interested. That's why the Open Luna Foundation is right up my alley.

This nonprofit project wants to use open source technologies and private donations to build a permanent outpost on the moon. (This isn't associated with the Google Lunar X Prize.)

The project's mission page details a five-visit gameplan. A couple things jumped out at me:

• They'll claim land around the moon's southern pole so they can eventually build a scientific outpost. This assumes they can get around extraterrestrial land issues.
• At least one communications satellite will be placed in orbit around the moon.
• I was struck by the group's entrepreneurial streak. Profits from moon collectibles (rocks, I'm presuming) will help fund future missions. Naming rights, licensed merchandise, contests and documentary films are also under consideration.
• Robots will handle early visits. Later, a single astronaut -- one who doesn't mind being alone -- will set up shop in a lunar "tent" and lay the groundwork for the outpost. A handful of astronauts will visit at a later date to expand construction.
• According to this slide deck (PDF), the manned outpost will take 5-7 years to construct and cost $500-$700 million.

The project's website and wiki have more details.

I'm curious what others think about this project. Can an open source mindset apply to huge ideas, like establishing a moon outpost? Please weigh in through the comments.

Four short links: 19 January 2010

Stack Overflow Data, Open Source GSM, Nostalgia, and Openness

1. Stack Overflow Data Dump -- all public data in Stack Overflow, Server Fault, and Super User.
2. OpenBTS -- an open-source Unix application that uses the Universal Software Radio Peripheral (USRP) to present a GSM air interface ("Um") to standard GSM handset and uses the Asterisk software PBX to connect calls. Portable mobile phone basestation that routes calls over the Internet.
3. Should We Encourage Self-Promotion and Lies? (Tom Coates) -- And while encouraging people to spot the talented and the creative, we should also be considering how we shame those people who self-promote without creating. The financial collapse has taught us that rhetorical bubbles divorced from reality are a danger to us all. We're already approaching this point - our industry has become venal, insular and dominated by marketing. We have come to value the wrong things. And if we want a continued vigorous, creative, free, open and equal environment, that's something we have to fix. It's not something to aspire to. Related: danah boyd's tweet, Sometimes I feel deeply nostalgic about the days when the interwebs were filled with the techno-utopian dreams of geeks and freaks.
4. The Opposite of Open is Theirs (David Weinberger) -- absolutely nails the nature of openness. A quick must-read. (via timo on Delicious)

Pew Research asks questions about the Internet in 2020

Will Google Make Us Stupid? Will we live in the cloud or the desktop?

Pew Research, which seems to be interested in just about everything, conducts a "future of the Internet" survey every few years in which they throw outrageously open-ended and provocative questions at a chosen collection of observers in the areas of technology and society. Pew makes participation fun by finding questions so pointed that they make you choke a bit. You start by wondering, "Could I actually answer that?" and then think, "Hey, the whole concept is so absurd that I could say anything without repercussions!" So I participated in their and did it again this week. The Pew report will aggregate the yes/no responses from the people they asked to participate, but I took the exercise as a chance to hammer home my own choices of issues.

(continue reading)

Four short links: 1 January 2010

Fonty Inkness, Machine Learning, Time-Series Indexes, and Graph Analysis

1. Measuring Type -- clever way to measure which font uses more ink.
2. Vowpal Wabbit -- fast learning software from Yahoo! Research and Hunch. Code available in git. (via zecharia on Delicious)
3. Literature Review on Indexing Time-Series Data -- a graduate student's research work included this literature review of papers on indexing time-series data. (via jpatanooga on Delicious)
4. igraph -- programming library for manipulating graph data, with the usual algorithms (minimum spanning tree, network flow, cliques, etc.) available in R, Python, and C.

Four short links: 30 November 2009

Paywall Performance, News Decisions, Sony Subsidising US Supercomputer, Invisible Open Source Business Model

1. Paywall Performance for News -- the National Business Review (NBR) in New Zealand went to a paywall in mid-July, and Foo Camper Lance Wiggs says their visitor numbers reveal a grim picture. As a commenter says, of course, visitor numbers go down but NBR makes money directly from the visitors that stay. I'm curious to see the effect on advertisers now the site's incentives are not to spray their load far and wide to land on as many eyeballs as possible. An interesting canary in the mine for Rupert's paywall plans at Fox.
2. Real Time, Real Discussion, Real Reporting: Choose Two (CrunchGear) -- a long post about the Internet's effects on journalism, but the headline will stick with me the longest.
3. Sony Still Subsidizing US Supercomputer Efforts -- US military buying PS3s as a cheap source of cell CPUs. The PS3's retail price is subsidized by Sony, driving game sales in a razor-blades model. It's like you could melt down razors and get more in scrap metal than they cost to buy at the supermarket ... (via BoingBoing)
4. Open Source Proves Elusive as Business Model (NYTimes) -- To Ms. Kroes’s point, there is an open-source alternative, and usually a pretty good one, to just about every major commercial software product. In the last decade, these open-source wares have put tremendous pricing pressure on their proprietary rivals. Governments and corporations have welcomed this competition. Whether open-source firms are practical as long-term businesses, however, is a much murkier question. On the contra side, Mozilla makes millions from referred searches and must be counted as a win for open source even though it's not a company.

Robots.Txt and the .Gov TLD

I'm on the board of CommonCrawl.Org, a nonprofit corporation that is attempting to provide a web crawl for use by all. An interesting report just got sent to us about the use of robots.txt files within the .Gov Top Level Domain, a standard known as the Robots Exclusion Standard.

In examining about 32,000 subdomains in .gov, it turns at least 1,188 of these have a robots.txt file with a "global disallow," meaning robots are excluded from indexing this content. Even more curious, on 175 of these sites, while there is a global disallow, there is a specific bypass that allows the Googlebot to index the data. You can look at the raw data on Factual.

At Public.Resource.Org, we've always felt that the use of a robots.txt file by the government should only be used for purposes of security and integrity of the site, not because some webmaster arbitrarily decides they don't want to be indexed. Indeed, on several occasions we have deliberately ignored government imposed robots.txt files because we felt this was an arbitrary and illegal attempt to keep the public out.

And, needless to say, it doesn't make any sense at all to let in some webcrawlers and not let in others. If this is a reaction to a security/integrity issue, such as limited capacity, the proper thing to do is include in the robots.txt file a comment that can be used by other bots to explain what is going on. For example, it could be perfectly reasonable for a government group faced with limited capacity to ask a robot to limit crawls to a certain number of queries per second and only whitelist crawlers that agree to that condition.

Government webmasters should use the robots.txt file sparingly, and should do so in a non-discriminatory fashion.

Four short links: 13 November 2009

Open Source Design, Interesting NoSQL Use, Copyright Documentary, Location Intelligence

1. Open Source Enters The World of Atoms -- an academic statistical analysis of open design. We indicated that, in open design communities, tangible objects can be developed in very similar fashion to software; one could even say that people treat a design as source code to a physical object and change the object via changing the source.
2. Why I Like Redis (Simon Willison) -- coherent explanation of why Simon likes and uses a particular nosql system. I can run a long running batch job in one Python interpreter (say loading a few million lines of CSV in to a Redis key/value lookup table) and run another interpreter to play with the data that’s already been collected, even as the first process is streaming data in. I can quit and restart my interpreters without losing any data. And because Redis semantics map closely to Python native data types, I don’t have to think for more than a few seconds about how I’m going to represent my data.
3. © kiwiright (Vimeo) -- short documentary about copyright, made to raise awareness of the issues in New Zealand. (just as applicable to the rest of the world)
4. Your Movements Speak For Themselves (Jeff Jonas) -- Mobile devices in America are generating something like 600 billion geo-spatially tagged transactions per day. Every call, text message, email and data transfer handled by your mobile device creates a transaction with your space-time coordinate (to roughly 60 meters accuracy if there are three cell towers in range), whether you have GPS or not. Got a Blackberry? Every few minutes, it sends a heartbeat, creating a transaction whether you are using the phone or not. If the device is GPS-enabled and you’re using a location-based service your location is accurate to somewhere between 10 and 30 meters. Using Wi-Fi? It is accurate below10 meters. A thought-provoking roundup of the information leakage with modern locative systems. (via TomC on Twitter)

Four short links: 12 November 2009

CRM on Rails, Data Mining on Hadoop, Disappointing Keynotes, The Teapot Effect

1. Fat Free CRM -- open source (Affero GPL) Ruby on Rails CRM system.
2. Bixo -- open source data mining toolkit that runs as a series of pipes on top of Hadoop. Built on Cascading workflow system for Hadoop that hides MapReduce. (via kdnuggets)
3. Andy Kessler's Keynote at Defrag Stank (Pete Warden) -- I'm sorry to hear it, because I loved Andy's book How We Got Here about the intersecting histories of economics, finance, and technology. Read the book instead of reading about the disappointing keynote.
4. The Teapot Effect -- the thing I love about geeks is how their passion causes them to explore, ruthlessly and quantitatively, the everyday phenomena that the rest of us take for granted. Such as dribbling teapots: “Previous studies have shown that dribbling is the result of flow separation where the layer of fluid closest to the boundary becomes detached from it. When that happens, the fluid flows smoothly over the lip. But as the flow rate decreases, the boundary layer re-attaches to the surface causing dribbling.” Read the post and the research it talks about to learn how to prevent Dribbling Teapot Syndrome ....

Recent Posts

• Four short links: 11 November 2009 | by Nat Torkington on November 11, 2009
• Unlikely Group Working Happily Together To Solve Patent Problem | by Carl Malamud on November 8, 2009
• Four short links: 4 November 2009 | by Nat Torkington on November 4, 2009
• Four short links: 27 October 2009 | by Nat Torkington on October 27, 2009
• Four short links: 26 October 2009 | by Nat Torkington on October 26, 2009
• Four short links: 23 October 2009 | by Nat Torkington on October 23, 2009
• Four short links: 7 October 2009 | by Nat Torkington on October 7, 2009
• Questions (and Answers!) About the Federal Register | by Carl Malamud on October 6, 2009
• Four short links: 5 October 2009 | by Nat Torkington on October 5, 2009
• Four short links: 29 September 2009 | by Nat Torkington on September 29, 2009
• Four short links: 24 September 2009 | by Nat Torkington on September 24, 2009
• Four short links: 18 September 2009 | by Nat Torkington on September 18, 2009