"scale" entries

Four short links: 11 August 2014

Startup Anthropology, Ends to Means, Permission to Test, and Distributed Systems Research

  1. Anthropology of Mid-Sized Startups — old but good post about the structures, norms, and dimensions of startup culture. Like a religion, a startup will care for its collective interest by defining certain things as sacred. A classic example is the company’s logo. This symbol is, quite literally, “set apart and forbidden” by brand guidelines, which often specify exactly how the logo must be presented and how far it should sit from the other elements on a page (thus separating the sacred from the profane).
  2. What Leads To — I love the elegant mechanic of decomposing an end back to a means you can do right now. Lots more sophistication obviously possible, but the fact that it’s not just about “thumbs up this end!” or about actions divorced from intention, makes it a step ahead for social software.
  3. Researching Link Rot (Pinboard) — graceful notification of a test, and with the simple ability to opt-out.
  4. The Space Between Theory and Practice in Distributed Systems (Marc Brooker) — I went through everything I’ve read on distributed systems and arranged them on a spectrum from theory to practice the two ends would be really well populated, but the middle would be disturbingly empty. Worse, changing to a graph of citation links would show a low density from theory to practice.
Comment
Four short links: 6 August 2014

Four short links: 6 August 2014

Mesa Database, Thumbstoppers, Impressive Research, and Microsoft Development

  1. Mesa: Geo-Replicated, Near Real-Time, Scalable Data Warehousing (PDF) — paper by Googlers on the database holding G’s ad data. Trillions of rows, petabytes of data, point queries with 99th percentile latency in the hundreds of milliseconds and overall query throughput of trillions of rows fetched per day, continuous updates on the order of millions of rows updated per second, strong consistency and repeatable query results even if a query involves multiple datacenters, and no SPOF. (via Greg Linden)
  2. Thumbstopping (Salon) — The prime goal of a Facebook ad campaign is to create an ad “so compelling that it would get people to stop scrolling through their news feeds,” reports the Times. This is known, in Facebook land, as a “thumbstopper.” And thus, the great promise of the digitial revolution is realized: The best minds of our generation are obsessed with manipulating the movement of your thumb on a smartphone touch-screen.
  3. om3d — pose a model based on its occurrence in a photo, then update the photo after rotating and re-rendering the model. Research is doing some sweet things these days—this comes hot on the heels of recovering sounds from high-speed video of things like chip bags.
  4. Microsoft’s Development Practices (Ars Technica) — they get the devops religion but call it “combined engineering”. They get the idea of shared code bases, but call it “open source”. At least when they got the agile religion, they called it that. Check out the horror story of where they started: a two-year development process in which only about four months would be spent writing new code. Twice as long would be spent fixing that code. MSFT’s waterfall was the equivalent of American football, where there’s 11 minutes of actual play in the average 3h 12m game.
Comment
Four short links: 5 August 2014

Four short links: 5 August 2014

Discussion Graph Tool, Superlinear Productivity, Go Concurrency, and R Map/Reduce Tools

  1. Discussion Graph Tool (Microsoft Research) — simplifies social media analysis by making it easy to extract high-level features and co-occurrence relationships from raw data.
  2. Superlinear Productivity in Collective Group Actions (PLoS ONE) — study of open source projects shows small groups exhibit non-linear productivity increases by size, which drop off at larger sizes. we document a size effect in the strength and variability of the superlinear effect, with smaller groups exhibiting widely distributed superlinear exponents, some of them characterizing highly productive teams. In contrast, large groups tend to have a smaller superlinearity and less variability.
  3. coop — cheat sheet of the most common concurrency program flows in Go.
  4. Tessera — set of open source tools around Hadoop, R, and visualization.
Comment
Four short links: 4 August 2014

Four short links: 4 August 2014

Web Spreadsheet, Correlated Novelty, A/B Ethics, and Replicated Data Structures

  1. EtherCalcopen source web-based spreadsheet.
  2. Dynamics of Correlated Novelties (Nature) — paper on “the adjacent possible”. Here we propose a simple mathematical model that mimics the process of exploring a physical, biological, or conceptual space that enlarges whenever a novelty occurs. The model, a generalization of Polya’s urn, predicts statistical laws for the rate at which novelties happen (Heaps’ law) and for the probability distribution on the space explored (Zipf’s law), as well as signatures of the process by which one novelty sets the stage for another. (via Steven Strogatz)
  3. On The Media Interview with OKCupid CEO — relevant to the debate on ethics of A/B tests. I preferred this to Tim Carmody’s rant.
  4. CRDTs as Alternative to APIswhen using CRDTs to tie your system together, you don’t need to resort to using impoverished representations that simply never come anywhere near the representational power of the data structures you use in your programs at runtime. See also this paper on Convergent and Commutative Replicated Data Types.
Comment
Four short links: 29 July 2014

Four short links: 29 July 2014

Community Detection, Proven Kernel, Graph Processing on GPUs, and Browser Vision

  1. Online Community Detection for Large Complex Networks (PLosONE) — readable recount of earlier algorithms and inventions in the area, as well as a new algorithm with linear time complexity for large complex networks.
  2. sel4 — open source OS kernel (GPLv2, most userland is BSD) with end-to-end proof of implementation correctness and security enforcement. (For a discussion of what’s verified, see this blog post)
  3. mapgraph.ioMassively Parallel Graph processing on GPUs. (via Leo Meyerovich)
  4. tracking.js — browser framework and algorithms for computer vision algorithms and frameworks.
Comment: 1
Four short links: 28 July 2014

Four short links: 28 July 2014

Secure Server, Angular Style, Recursion History (see Recursion History), Aerospike Open Source

  1. streisandsets up a new server running L2TP/IPsec, OpenSSH, OpenVPN, Shadowsocks, Stunnel, and a Tor bridge. It also generates custom configuration instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists.
  2. Angular.js Style Guidemy opinionated styleguide for syntax, building and structuring Angular applications.
  3. How Recursion Got into ProgrammingCommittee member F.L. Bauer registered his protest by characterizing the addition of recursion to the language as an “Amsterdam plot”.
  4. aerospike — open source database server and client, with bold claims of performance.
Comment
Four short links: 22 July 2014

Four short links: 22 July 2014

English lint, Scalable Replicated Datastore, There's People in my Software, and Sci-Fi for Ethics

  1. write-gooda naive `lint’ for English prose.
  2. cockroachdba scalable, geo-replicated, transactional datastore from a team that includes the person who built Spanner for Google. Spanner requires atomic clocks, cockroach does not (which has corresponding performance consequences). (via Wired)
  3. The Deep Convergence of Networks, Software, and Peopleas we wire up our digital products increasingly with interconnected networks, their nature is increasingly a product of the responses that come back from those networks. The experience cannot be wholly represented in mock prototypes that are coded to respond in predictable ways, or even using a set of preset random responses. The power of the application is seeing the emergent behaviour of the system, and recognizing that you are a participant in that emergent behaviour. (via Tim O’Reilly)
  4. An Ethics Class for Inventors, via Sci-Fi“Reading science fiction is kind of like ethics class for inventors,” says Brueckner. Traditionally, technology schools ask ‘how do we build it?’ This class asks a different question: ‘should we?’
Comments: 2
Four short links: 8 July 2014

Four short links: 8 July 2014

Virtual Economies, Resource UAVs, Smarter Smaller Crowds, and Scaling Business

  1. Virtual Economies — new book from MIT Press on economics in games. The book will enable developers and designers to create and maintain successful virtual economies, introduce social scientists and policy makers to the power of virtual economies, and provide a useful guide to economic fundamentals for students in other disciplines.
  2. Resource Industry UAV Conference Presentations — collection of presentations from a recent resources industry conference. Includes UaaS: UAVs as a Service. (via DIY Drones)
  3. The Wisdom of Smaller, Smarter Crowdsin domains in which some crowd members have demonstrably more skill than others, smart sub-crowds could possibly outperform the whole. The central question this work addresses is whether such smart subsets of a crowd can be identified a priori in a large-scale prediction contest that has substantial skill and luck components. (via David Pennock)
  4. Larry and Sergey with Vinod (YouTube) — see transcription. I really liked Page’s point about scaling the number of things that companies do, and the constraints on such scaling.
Comment

Four short links: 4 July 2014

Deleted Transparency, Retro Theme, MPA Suckage, and Ultrasonic Comms

  1. The Flipside of the Right To Be Forgotten (Business Insider) — deletion requests were granted for a former politician who wanted to remove links to a news article about his behavior when previously in office – so that he can have a clean slate when running for a new position – and a man who was convicted of possessing child sexual abuse imagery.
  2. BOOTSTRA.386 — gorgeously retro theme for Bootstrap.
  3. Multi-Process Architectures Suck — detailed and painful look at the computational complexity and costs of multiprocess architectures.
  4. Chromecast Ultrasonic CommsIn the new system, Chromecast owners first allow support for nearby devices. A nearby device then requests access to the Chromecast, and the Chromecast plays an ultrasonic sound through the connected TV’s speakers. The sound is then picked up by the microphone in the device, which allows it to pair with the TV. (via Greg Linden)
Comment
Four short links: 15 May 2014

Four short links: 15 May 2014

Pervasive Monitoring, Mozilla DRM, Game Finances, and Distributed Systems

  1. Pervasive Monitoring is an Attack (Tim Bray) — if your ap­pli­ca­tion doesn’t sup­port pri­va­cy, that’s prob­a­bly a bug in your ap­pli­ca­tion.
  2. Reconciling Mozilla’s Mission and the W3C EME — essentially, “we don’t want to put a closed source bolus of evil into our open source unicorn, but you won’t be able to watch House of Cards with Firefox if we don’t.”
  3. The Financial Future of Game Developers (Raph Koster) — Today, a console is really just a hardware front end to a digital publisher/distribution network/storefront. […] Any structure that depends solely on blockbusters is not long for this world, because there is a significant component of luck in what drives popularity, so every release is literally a gamble. […] The median game uploaded to the App Store makes zero dollars. It starts great and just gets better. Koster is on fire! He scores again! GOOOOOOOOOOOOOOOAL!
  4. Notes on Distributed Systems for Young Bloods“It’s slow” is the hardest problem you’ll ever debug.
Comment