- Accumulo — NSA’s BigTable implementation, released as an Apache project.
- How the Robots Lost (Business Week) — the decline of high-frequency trading profits (basically, markets worked and imbalances in speed and knowledge have been corrected). Notable for the regulators getting access to the technology that the traders had: Last fall the SEC said it would pay Tradeworx, a high-frequency trading firm, $2.5 million to use its data collection system as the basic platform for a new surveillance operation. Code-named Midas (Market Information Data Analytics System), it scours the market for data from all 13 public exchanges. Midas went live in February. The SEC can now detect anomalous situations in the market, such as a trader spamming an exchange with thousands of fake orders, before they show up on blogs like Nanex and ZeroHedge. If Midas sees something odd, Berman’s team can look at trading data on a deeper level, millisecond by millisecond.
- PRISM: Surprised? (Danny O’Brien) — I really don’t agree with the people who think “We don’t have the collective will”, as though there’s some magical way things got done in the past when everyone was in accord and surprised all the time. It’s always hard work to change the world. Endless, dull hard work. Ten years later, when you’ve freed the slaves or beat the Nazis everyone is like “WHY CAN’T IT BE AS EASY TO CHANGE THIS AS THAT WAS, BACK IN THE GOOD OLD DAYS. I GUESS WE’RE ALL JUST SHEEPLE THESE DAYS.”
- What We Don’t Know About Spying on Citizens is Scarier Than What We Do Know (Bruce Schneier) — The U.S. government is on a secrecy binge. It overclassifies more information than ever. And we learn, again and again, that our government regularly classifies things not because they need to be secret, but because their release would be embarrassing. Open source BigTable implementation: free. Data gathering operation around it: $20M/year. Irony in having the extent of authoritarian Big Brother government secrecy questioned just as a whistleblower’s military trial is held “off the record”: priceless.
Open Source BigTable, Robots Lost, Changing the World, Secrecy Binge
- World History Since 1300 (Coursera) — Coursera expands offerings to include humanities. This content is in books and already in online lectures in many formats. What do you get from these? Online quizzes and the online forum with similar people considering similar things. So it’s a book club for a university course?
- mod_spdy — Apache module for the SPDY protocol, Google’s “faster than HTTP” HTTP.
- The Top 10 Dying Industries in the United States (Washington Post) — between the Internet and China, yesterday’s cash cows are today’s casseroles.
Doug Cutting on Hadoop's rise and why he's surprised at its growth.
Doug Cutting discusses Hadoop's current and near-term role, and the factors that made it a central part of data processing.
It's unlikely IBM or Apache will lead the Java community.
Why did Mike Loukides leave IBM and Apache out of his recent piece, “Who leads the Java Parade?” Because — despite good reasons — they both opted out.
Historic Debt, Historic Naming, Autonomous Quadcopter, and Entrepreneurial Thought
- Debt: The First 5,000 Years — Throughout its 5000 year history, debt has always involved institutions – whether Mesopotamian sacred kingship, Mosaic jubilees, Sharia or Canon Law – that place controls on debt’s potentially catastrophic social consequences. It is only in the current era, writes anthropologist David Graeber, that we have begun to see the creation of the first effective planetary administrative system largely in order to protect the interests of creditors. (via Tim O’Reilly)
- Know Your History — where Google’s +1 came from (answer: Apache project).
- MIT Autonomous Quadcopter — MIT drone makes a map of a room in real time using an X Box Kinect and is able to navigate through it. All calculations performed on board the multicopter. Wow. (via Slashdot and Sara Winge)
- How Great Entrepreneurs Think — leaving aside the sloppy open-mouth kisses to startups that “great entrepreneurs” implies, an interesting article comparing the mindsets of corporate execs with entrepreneurs. I’d love to read the full interviews and research paper. Sarasvathy explains that entrepreneurs’ aversion to market research is symptomatic of a larger lesson they have learned: They do not believe in prediction of any kind. “If you give them data that has to do with the future, they just dismiss it,” she says. “They don’t believe the future is predictable…or they don’t want to be in a space that is very predictable.” […] the careful forecast is the enemy of the fortuitous surprise. (via Sacha Judd)
Storage, MapReduce and Query are ushering in data-driven products and services.
We're at the beginning of a revolution in data-driven products and services, driven by a software stack that enables big data processing on commodity hardware. Learn about the SMAQ stack, and where today's big data tools fit in.
Reading Outlook in Open Source, Android Tablets, Websocket Editing, Jabber for Node.js
- PSTSDK — Apache-licensed code from Microsoft to read Outlook files. Covered by Microsoft’s Open Specification Promise not to assert related patents against users of this library.
- Cheap Android Tablet — not multitouch, but only $136. Good for hacking with in the meantime. (via Hacker News)
- Real-Time Collaborative Editing with Websockets, node.js, and Redis — uses Chrome’s websockets alternative to Comet and other long-polling web connections.