ENTRIES TAGGED "devops"

Exploring lightweight monitoring systems

Toward unifying customer behavior and operations metrics.

lightweight_systemsFor the last ten years I’ve had a foot in both the development and operations worlds. I stumbled into the world of IT operations as a result of having the most UNIX skills in the team shortly after starting at ThoughtWorks. I was fortunate enough to do so at a time when many of my ThoughtWorks colleagues and I where working on the ideas which were captured so well in Jez Humble and Dave Farley’s Continuous Delivery (Addison-Wesley).

During this time, our focus was on getting our application into production as quickly as possible. We were butting up against the limits of infrastructure automation and IaaS providers like Amazon were only in their earliest form.

Recently, I have spent time with operations teams who are most concerned with the longer-term challenges of looking after increasingly complex ecosystems of systems. Here the focus is on immediate feedback and knowing if they need to take action. At a certain scale, complex IT ecosystems can seem to exhibit emergent behavior, like an organism. The operations world has evolved a series of tools which allow these teams to see what’s happening *right now* so we can react, keep things running, and keep people happy.

At the same time, those of us who spend time thinking about how to quickly and effectively release our applications have become preoccupied with wanting to know if that software does what our customers want once it gets released. The Lean Startup movement has shown us the importance of putting our software in front of our customers, then working out how they actually use it so we can determine what to do next. In this world, I was struck by the shortcomings of the tools in this space. Commonly used web analytics tools, for example, might only help me understand tomorrow how my customers used my site today.

Read more…

Comment
Four short links: 29 May 2014

Four short links: 29 May 2014

Modern Software Development, Internet Trends, Software Ethics, and Open Government Data

  1. Beyond the Stack (Mike Loukides) — tools and processes to support software developers who are as massively distributed as the code they build.
  2. Mary Meeker’s Internet Trends 2014 (PDF) — the changes on slide 34 are interesting: usage moving away from G+/Facebook-style omniblather creepware and towards phonebook-based chat apps.
  3. Introduction to Software Engineering Ethics (PDF) — amazing set of provocative questions and scenarios for software engineers about the decisions they made and consequences of their actions. From a course in ethics from SCU.
  4. Open Government Data Online: Impenetrable (Guardian) — Too much knowledge gets trapped in multi-page pdf files that are slow to download (especially in low-bandwidth areas), costly to print, and unavailable for computer analysis until someone manually or automatically extracts the raw data.
Comment
Four short links: 8 May 2014

Four short links: 8 May 2014

OpenStack Critique, Good Teaching, More Good Teaching, and Programming Sucks

  1. OpenStack: A Plea — critical take on OpenStack, also in this presentation. Notes the proliferation of inefficiency, devops bolted onto the side, and the long feedback cycle. You code differently when you have a pager. (via Sam Ramji)
  2. What Could Be More Interesting Than How The Mind Works? (Harvard) — the Steven Pinker story. A third ingredient of good teaching is overcoming “the curse of knowledge”: the inability to know what it’s like not to know something that you do know. (via Atul Gawande)
  3. Some Lecturing HeuristicsRealize further that your mood may be determined by only a few people. A smiling nodder will make you feel good, and you will do better. People reading newspapers will make you feel bad, and you will do worse. Do not permit people to do things that make you feel bad.
  4. Programming Sucks — every word is true.
Comment
Four short links: 30 April 2014

Four short links: 30 April 2014

Critical Making, Torrent Filesystem, Testing Infrastructure, and Reproducible Research

  1. Critical Making — essays from 70 contributors looking at the politics, choices, and ethics of a lot of the makery going on.
  2. torrent-mount — mount a torrent as a filesystem in real time using Javascript. (via Joe McCann)
  3. Continuous Integration for Infrastructure — slides on the emerging tools for large-scale automated testing integrated into development and deployment workflow.
  4. Implementing Reproducible Research — book by Victoria Stodden and Johanna Cohoon on tools, practices, and platforms for making science that others can verify (another step in improving velocity and quality of scientific research).
Comment
Four short links: 14 April 2014

Four short links: 14 April 2014

dategrep, Agile Signoff, Feedback Speed, and Modern Dev

  1. dategrepprint lines matching ranges of dates. Genius!
  2. Business Case Guidance in Agile Projects (gov.uk) — how the UK govt signs off on Agile projects, which normally governments have no clue over how to handle properly.
  3. Hyper Growth Done Right“While I was at Oracle, it took a month before a new engineer would get any code in,” Agarwal says. “It sent this implicit message that it’s okay to take a month to write some code.” First time I’d heard this wise point articulated: slow feedback loops send the message that progress can be slow.
  4. Docker + Github + Jenkins — clever integration of the three tools to get repeatable continuous integration. The modern dev environment has workflow built on git, VMs, and glue.
Comment

Full-stack developers

Developers who understand the whole stack are going to build better applications.

Large tree with branches. Photo by Alex, used under a Creative Commons license

Some see the full-stack developer as a unicorn, but it’s starting to look more like a tree, with tooling, cloud services, design, data, and networking added.

Since Facebook’s Carlos Bueno wrote the canonical article about the full stack, there has been no shortage of posts trying to define it. For a time, Facebook allegedly only hired “full-stack developers.” That probably wasn’t quite true, even if they thought it was. And some posts really push “full-stack” developer into Unicorn territory: Laurence Gellert writes that it “goes beyond being a senior engineer,” and details everything he thinks a full-stack developer should be familiar with, most of which doesn’t involve coding. Read more…

Comment: 1
Four short links: 28 March 2014

Four short links: 28 March 2014

Javascript on Glass, Smart Lights, Hardware Startups, MySQL at Scale

  1. WearScript — open source project putting Javascript on Glass. See story on it. (via Slashdot)
  2. Mining the World’s Data by Selling Street Lights and Farm Drones (Quartz) — Depending on what kinds of sensors the light’s owners choose to install, Sensity’s fixtures can track everything from how much power the lights themselves are consuming to movement under the post, ambient light, and temperature. More sophisticated sensors can measure pollution levels, radiation, and particulate matter (for air quality levels). The fixtures can also support sound or video recording. Bring these lights onto city streets and you could isolate the precise location of a gunshot within seconds.
  3. An Investor’s Guide to Hardware Startups — good to know if you’re thinking of joining one, too.
  4. WebScaleSQL — a MySQL downstream patchset built for “large scale” (aka Google, Facebook type loads).
Comment
Four short links: 3 March 2014

Four short links: 3 March 2014

Vanishing Money, Car Hackery, Data Literacy Course, and Cheaper CI

  1. The Programming Error That Cost Mt Gox 2609 Bitcoins — in the unforgiving world of crypto-currency, it’s easy to miscode and vanish your money.
  2. Ford Invites Open-Source Community to Tinker AwayOne example: Nelson has re-tasked the motor from a Microsoft Xbox 360 game controller to create an OpenXC shift knob that vibrates to signal gear shifts in a standard-transmission Mustang. The 3D-printed prototype shift knob uses Ford’s OpenXC research platform to link devices to the car via Bluetooth, and shares vehicle data from the on-board diagnostics port. Nelson has tested his prototype in a Ford Mustang Shelby GT500 that vibrates at the optimal time to shift.
  3. Making Sense of Data — Google online course on data literacy.
  4. Cost-Efficient Continuous Integration at Mozilla — CI on a big project can imply hundreds if not thousands of VMs on Amazon spinning up to handle compiles and tests. This blog post talks about Mozilla’s efforts to reduce its CI-induced spend without reducing the effectiveness of its CI practices.
Comment
Four short links: 28 February 2014

Four short links: 28 February 2014

Minecraft+Pi+Python, Science Torrents, Web App Performance Measurement, and Streaming Data

  1. Programming Minecraft Pi with Python — an early draft, but shows promise for kids. (via Raspberry Pi)
  2. Terasaur — BitTorrent for mad-large files, making it easy for datasets to be saved and exchanged.
  3. BuckyOpen-source tool to measure the performance of your web app directly from your users’ browsers. Nifty graph.
  4. Zoe Keating’s Streaming Payouts — actual data on a real musician’s distribution and revenues through various channels. Hint: streaming is tragicomically low-paying. (via Andy Baio)
Comment
Four short links: 27 February 2014

Four short links: 27 February 2014

A Fine Rant, Continuous Deployment, IBeacon Spec, and LaTeX Gets a Collaborative Multiplayer Mode

  1. Our Comrade, The Electron (Maciej Ceglowski) — a walk through the life of the inventor of the Theremin, with a pointed rant about how we came to build the surveillance state for the state. One of the best conference talks ever, and I was in the audience for it!
  2. go.cd — continuous deployment and delivery automation tool from Thoughtworks, nothing to do with the Go programming language. The name is difficult to search for, so naturally we needed the added confusion of two projects sharing the name. Continuous deployment is an important part of devops (“the job of our programmers is not to write code, it is to deploy working code into production”—who said this? I’ve lost the reference already).
  3. Apple iBeacon Developer Programme — info locked up behind registration. Sign an NDA to get the specs, free to use the name. Interesting because iBeacon and other Bluetooth LE implementations are promising steps to building a network of things. (via Beekn)
  4. ShareLaTeX — massively multiplayer online LaTeX. Open sourced.
Comment