- Seagate Kinetic Storage — In the words of Geoff Arnold: The physical interconnect to the disk drive is now Ethernet. The interface is a simple key-value object oriented access scheme, implemented using Google Protocol Buffers. It supports key-based CRUD (create, read, update and delete); it also implements third-party transfers (“transfer the objects with keys X, Y and Z to the drive with IP address 1.2.3.4”). Configuration is based on DHCP, and everything can be authenticated and encrypted. The system supports a variety of key schemas to make it easy for various storage services to shard the data across multiple drives.
- Masters of Their Universe (Guardian) — well-written and fascinating story of the creation of the Elite game (one founder of which went on to make the Raspberry Pi). The classic action game of the early 1980s – Defender, Pac Man – was set in a perpetual present tense, a sort of arcade Eden in which there were always enemies to zap or gobble, but nothing ever changed apart from the score. By letting the player tool up with better guns, Bell and Braben were introducing a whole new dimension, the dimension of time.
- Micropolar (github) — A tiny polar charts library made with D3.js.
- Introduction to R (YouTube) — 21 short videos from Google.
"storage" entries
Full-stack tensions on the Web
How much do you need to know?
I expected that CSSDevConf would be primarily a show about front-end work, focused on work in clients and specifically in browsers. I kept running into conversations, though, about the challenges of moving between the front and back end, the client and the server side. Some were from developers suddenly told that they had to become “full-stack developers” covering the whole spectrum, while others were from front-end engineers suddenly finding a flood of back-end developers tinkering with the client side of their applications. “Full-stack” isn’t always a cheerful story.
In the early days of the Web, “full-stack” was normal. While there were certainly people who focused on running web servers or designing sites as beautiful as the technology would allow, there were lots of webmasters who knew how to design a site, write HTML, manage a server, and maybe write some CGI code for early applications.
Formal separation of concerns among HTML, CSS, and JavaScript made it easier to share responsibilities among specialists. As the dot-com boom proceeded, specialization accelerated, with dedicated designers, programmers, and sysadmins coming to the work. Perhaps there were too many titles.
Even as the bust set in, specialization remained the trend because Web projects — especially on the server side — had grown far more complicated. They weren’t just a server and a few scripts, but a complete stack, including templates, logic, and usually a database. Whether you preferred the LAMP stack, a Microsoft ASP stack, or perhaps Java servlets and JSP, the server side rapidly became its own complex arena. Intranet development in particular exploded as a way to build server-based applications that could (cheaply) connect data sources to users on multiple platforms. Writing web apps was faster and cheaper than writing desktop apps, with more tolerance for platform variation.
Read more…
Four short links: 25 October 2013
Disk Over Ethernet, Inside Elite, Polar Charts, and R Videos
An update on in-memory data management
In-memory data management brings data close to the computation.
By Ben Lorica and Roger Magoulas
We wanted to give you a brief update on what we’ve learned so far from our series of interviews with players and practitioners in the in-memory data management space. A few preliminary themes have emerged, some expected, others surprising.
Performance improves as you put data as close to the computation as possible. We talked to people in systems, data management, web applications, and scientific computing who have embraced this concept. Some solutions go to the the lowest level of hardware (L1, L2 cache), The next generation SSDs will have latency performance closer to main memory, potentially blurring the distinction between storage and memory. For performance and power consumption considerations we can imagine a future where the primary way systems are sized will be based on the amount of non-volatile memory* deployed.
Putting data in-memory does not negate the importance of distributed computing environments. Data size and the ability to leverage parallel environments are frequently cited reasons. The same characteristics that make the distributed environments compelling also apply to in-memory systems: fault-tolerance and parallelism for performance. An additional consideration is the ability to gracefully spillover to disk when main is memory full. Read more…
Complexity fails: A lesson from storage simplification
Storage architectures show simplicity's power and how to build clouds at scale.
Simple systems scale effectively, while complex systems struggle to overcome the multiplicative effect of potential failure points. This shows us why the most reliable and scalable clouds are those made up of fewer, simpler parts.
Four short links: 28 December 2011
Text Search, Cloud Filesystem, Javascript Parser, and Twitter Templates
- Terrier IR — open source (Mozilla) text search engine, now with Hadoop support.
- s3ql — open source (GPLv3) Linux filesystem which stores its data on Google Storage, Amazon S3, or OpenStack. (via Adam Shand)
- Esprima — open source (BSD) fast Javascript parser in Javascript. (via Javascript Weekly)
- Hogan.js — open source (Apache) Javascript templating engine from Twitter. If it proves anywhere near as good as Bootstrap, it’ll be heavily used.
Four short links: 16 July 2010
GPL Debate, Storage Costs, Social Software, Vodafone's Open Source
- GPL WordPress Theme Angst — a podcaster brought together Matt Mullenweg (creator of WordPress), and Chris Pearson (creator of the Thesis theme). Chris doesn’t believe WordPress’s GPL should be inherited by themes. Matt does, and the SFLC and others agree. The conversation is interesting because (a) they and the podcaster do a great job of keeping it civil and on-track and purposeful, and (b) Chris is unswayed. Chris built on GPLed software without realizing it, and is having trouble with the implications. Chris’s experience, and feelings, and thought processes, are replicated all around the world. This is like a usability bug for free software. (via waxpancake on Twitter)
- 480G SSD Drive — for a mere $1,599.99. If you wonder why everyone’s madly in love with parallel, it’s because of this order-of-magnitude+ difference in price between regular hard drives and the Fast Solution. Right now, the only way to rapidly and affordably crunch a ton of data is to go parallel. (via marcoarment on Twitter)
- Pandas and Lobsters: Why Google Cannot Build Social Software — this resonates with me. The primary purpose of a social application is connecting with others, seeing what they’re up to, and maybe even having some small, fun interactions that though not utilitarian are entertaining and help us connect with our own humanity. Google apps are for working and getting things done; social apps are for interacting and having fun. Read it for the lobster analogy, which is gold.
- Wayfinder — The majority of all the location and navigation related software developed at Wayfinder Systems, a fully owned Vodafone subsidiary, is made available publicly under a BSD licence. This includes the distributed back-end server, tools to manage the server cluster and map conversion as well as client software for e.g. Android, iPhone and Symbian S60. Technical documentation is available in the wiki and discussions around the software are hosted in the forum. Interesting, and out of the blue. At the very least, there’s some learning to be done by reading the server infrastructure. (via monkchips on Twitter)
Four short links: 21 December 2009
Social Networking Data, Memory Tech, Sim News, Microbe Jalopy Breakthrough
- A Taxonomy of Social Networking Data (Bruce Schneier) — he divides information by who gave it, why, and who controls it. Useful to remember that not all social data are equal.
- Five Ways to Revolutionise Computer Memory (New Scientist) — the physics and economics of new memory technology.
- News at Seven — project to automatically generate news report, complete with Flash-animated news readers and text-to-speech voices. A project from the Intelligent Information Lab at Northwestern University.
- Bacteria-Powered Micro-Machines — A few hundred bacteria are working together in order to turn the gear. When multiple gears are placed in the solution with the spokes connected like in a clock, the bacteria will begin turning both gears in opposite directions and it will cause the gears to rotate in synchrony for a long time. Video embedded below (via BoingBoing)
Four short links: 29 October 2009
Learning Programming, Functional Javascript, Controlling Firefox, Kicking Ass (with SSDs)
- Julie Learns to Program — blog from our own Julie Steele as she learns her first programming language. The point is: it’s in me. I wasn’t sure that is was, and now I know—it is. And what, exactly, is “it”? It is the bug. It is the combination of native curiosity and stubbornness that made me play around with the code and take some wild guesses instead of running straight to Google (or choosing to stay within the bounds of the exercise). That might sound like a small thing, but I know it is not. I was determined to make the program do what I wanted it to do, I came up with a few guesses as to how to do that, and I kept trying different things until I succeeded (and then I felt thrilled). As much as I have to learn, I know now that I really am hooked. And that I’ll get there.
- underscore.js — new Javascript library of functional programming primitives (map, each, inject, etc.). (via Simon Willison)
- WWW::Mechanize::Firefox — Perl module to control Firefox, using the same interface as the WWW::Mechanize web robot module. (via straup on Delicious)
- Anatomy of SSDs — teeth-rattlingly technical Linux Magazine article explaining the different types of SSDs (Solid State Disks–imagine a hard drive made of rapid-access Flash memory). Artur Bergman told me that installing an SSD drive in his MacBook Pro gave the greatest performance increase of any computer upgrade he’d performed since he went from no computer to one.
Four short links: 3 September 2009
Smarter Eyes, Urinal Protocol Efficiency, Petabytes on a Budget, and LocaLondon
- Many Eyes Make All Bugs Shallow, Especially When The Eyes Get Smarter (David Eaves) — Mozilla released bug submission data, and David realizes with some minor investment (particularly some simpler vetting screens prior to reaching bugzilla) bug submitters could learn faster. For example, a landing screen that asks you if you’ve ever submitted a bug before might take newbies to a different page where the bugzilla process is explained in greater detail, the fact that this is not a support site is outlined, and some models of good “submissions” are shared (along with some words of encouragement). By segmenting newbies we might ease the work burden on those who have to vet the bugs.
- Urinal Protocol Efficiency (xkcd blog) — geeks are pattern-matching creatures that can count. This leads us to a question: what is the general formula for the number of guys who will fill in N urinals if they all come in one at a time and follow the urinal protocol? One could write a simple recursive program to solve it, placing one guy at a time, but there’s also a closed-form expression. If f(n) is the number of guys who can use n urinals, f(n) for n>2 is given by: […] The protocol is vulnerable to producing inefficient results for some urinal counts. Some numbers of urinals encourage efficient packing, and others encourage sparse packing. (via Hacker News)
- Petabytes on a Budget: 67Tb for $7,867 — DIY cloud hardware. (via timhaines on Twitter)
- LocaLondon (Chris Heathcote) — informative, ingenious, and replicable (like all that Chris does), it’s a Twitter feed of art exhibitions in London (when they open, when there’s a week left, and on the last day) and a glorious horizontal touchscreen-friendly meta-reviews site so you can quickly see at a glance what’s on now and what people think of it.
Four short links: 4 August 2009
NASA Cloudware, btrfs, eBook Editing, Exponential Death
- NASA Nebula Services/Platform Stack — The NEBULA platform offers a turnkey Software-as-a-Service experience that can rapidly address the requirements of a large number of projects. However, each component of the NEBULA platform is also available individually; thus, NEBULA can also serve in Platform-as-a-Service or Infrastructure-as-a-Service capacities. Bundles RabbitMQ, Eucalyptus, LUSTRE storage, Fabric deployment, Varnish front-end, MySQL and more. (via Jim Stogdill)
- A Short History of btrfs — Now for some personal predictions (based purely on public information – I don’t have any insider knowledge). Btrfs will be the default file system on Linux within two years. Btrfs as a project won’t (and can’t, at this point) be canceled by Oracle. If all the intellectual property issues are worked out (a big if), ZFS will be ported to Linux, but it will have less than a few percent of the installed base of btrfs. Check back in two years and see if I got any of these predictions right!
- Sigil — open source WYSIWYG eBook editor. (via liza on Twitter)
- Exponential Decay of Life — This startling fact was first noticed by the British actuary Benjamin Gompertz in 1825 and is now called the “Gompertz Law of human mortality.” Your probability of dying during a given year doubles every 8 years. For me, a 25-year-old American, the probability of dying during the next year is a fairly miniscule 0.03% — about 1 in 3,000. When I’m 33 it will be about 1 in 1,500, when I’m 42 it will be about 1 in 750, and so on. (via Hacker News)