"metrics" entries

Four short links: 19 March 2015

Four short links: 19 March 2015

Changing Behaviour, Building Filters, Public Access, and Working Capital

  1. Using Monitoring Dashboards to Change Behaviour[After years of neglect] One day we wrote some brittle Ruby scripts that polled various services. They collated the metrics into a simple database and we automated some email reports and built a dashboard showing key service metrics. We pinpointed issues that we wanted to show people. Things like the login times, how long it would take to search for certain keywords in the app, and how many users were actually using the service, along with costs and other interesting facts. We sent out the link to the dashboard at 9am on Monday morning, before the weekly management call. Within 2 weeks most problems were addressed. It is very difficult to combat data, especially when it is laid out in an easy to understand way.
  2. Quiet Mitsubishi Cars — noise-cancelling on phone calls by using machine learning to build the filters.
  3. NSF Requiring Public AccessNSF will require that articles in peer-reviewed scholarly journals and papers in juried conference proceedings or transactions be deposited in a public access compliant repository and be available for download, reading, and analysis within one year of publication.
  4. Filtered for Capital (Matt Webb) — It’s important to get a credit line [for hardware startups] because growing organically isn’t possible — even if half your sell-in price is margin, you can only afford to grow your batch size at 50% per cycle… and whether it’s credit or re-investing the margin, all that growth incurs risk, because the items aren’t pre-sold. There are double binds all over the place here.
Comment
Four short links: 11 March 2015

Four short links: 11 March 2015

Working Manager, Open Source Server Chassis, Data Context, and Coevolved Design & Users

  1. As a Working Manager (Ian Bicking) — I look forward to every new entry in Ian’s diary, and this one didn’t disappoint. But I’m a working manager. Is now the right time to investigate that odd log message I’m seeing, or to think about who I should talk to about product opportunities? There’s no metric to compare the priority of two tasks that are so far apart. If I am going to find time to do development I am a bit worried I have two options: (1) Keep doing programming after hours; (2) Start dropping some balls as a manager.
  2. Introducing Yosemite (Facebook) — a modular chassis that contains high-powered system-on-a-chip (SoC) processor cards.
  3. The Joyless World of Data-Driven StartupsThere is so much invisible, fluid context wrapped around a data point that we are usually unable to fully comprehend exactly what that data represents or means. We often think we know, but we rarely do. But we really WANT it to mean something, because using data in our work is scientific. It’s not our decision that was wrong — we used the data that was available. Data is the ultimate scapegoat.
  4. History of the Urban Dashboardthe dashboard and its user had to evolve in response to one another. The increasing complexity of the flight dashboard necessitated advanced training for pilots — particularly through new flight simulators — and new research on cockpit design.
Comment
Four short links: 7 January 2015

Four short links: 7 January 2015

Program Synthesis, Data Culture, Metrics, and Information Biology

  1. Program Synthesis ExplainedThe promise of program synthesis is that programmers can stop telling computers how to do things, and focus instead on telling them what they want to do. Inductive program synthesis tackles this problem with fairly vague specifications and, although many of the algorithms seem intractable, in practice they work remarkably well.
  2. Creating a Data-Driven Culture — new (free!) ebook from Hilary Mason and DJ Patil. The editor of that team is the luckiest human being alive.
  3. Ev Williams on Metrics — a master-class in how to think about and measure what matters. If what you care about — or are trying to report on — is impact on the world, it all gets very slippery. You’re not measuring a rectangle, you’re measuring a multi-dimensional space. You have to accept that things are very imperfectly measured and just try to learn as much as you can from multiple metrics and anecdotes.
  4. Nature, the IT Wizard (Nautilus) — a fun walk through the connections between information theory, computation, and biology.
Comment
Four short links: 23 December 2014

Four short links: 23 December 2014

Useful Metrics, Trouble at Mill, Drug R&D, and Disruptive Opportunities

  1. Metrics for Operational Performance — you’d be surprised how many places around your business you can meaningfully and productively track time-to-detection and time-to-resolution.
  2. Steel Mill Hacked — damage includes a blast furnace that couldn’t be shut down properly.
  3. Cerebros — drug-smuggling’s equivalent of corporate R&D. (via Regine Debatty)
  4. Ramble About Bitcoin (Matt Webb) — the meta I’m trying to figure out is: when you spot that one of these deep value chains is at the beginning of a big reconfiguration, what do you do? How do you enter it as a small business? How, as a national economy, do you help it along and make sure the transition happens healthily?
Comment
Four short links: 14 July 2014

Four short links: 14 July 2014

Scanner Malware, Cognitive Biases, Deep Learning, and Community Metrics

  1. Handheld Scanners Attack — shipping and logistics operations compromised by handheld scanners running malware-infested Windows XP.
  2. Adventures in Cognitive Biases (MIT) — web adventure to build your cognitive defences against biases.
  3. Quoc Le’s Lectures on Deep Learning — Machine Learning Summer School videos (4k!) of the deep learning lectures by Google Brain team member Quoc Le.
  4. FLOSS Community Metrics Talks — upcoming event at Puppet Labs in Portland. I hope they publish slides and video!
Comment: 1

Exploring lightweight monitoring systems

Toward unifying customer behavior and operations metrics.

lightweight_systemsFor the last ten years I’ve had a foot in both the development and operations worlds. I stumbled into the world of IT operations as a result of having the most UNIX skills in the team shortly after starting at ThoughtWorks. I was fortunate enough to do so at a time when many of my ThoughtWorks colleagues and I where working on the ideas which were captured so well in Jez Humble and Dave Farley’s Continuous Delivery (Addison-Wesley).

During this time, our focus was on getting our application into production as quickly as possible. We were butting up against the limits of infrastructure automation and IaaS providers like Amazon were only in their earliest form.

Recently, I have spent time with operations teams who are most concerned with the longer-term challenges of looking after increasingly complex ecosystems of systems. Here the focus is on immediate feedback and knowing if they need to take action. At a certain scale, complex IT ecosystems can seem to exhibit emergent behavior, like an organism. The operations world has evolved a series of tools which allow these teams to see what’s happening *right now* so we can react, keep things running, and keep people happy.

At the same time, those of us who spend time thinking about how to quickly and effectively release our applications have become preoccupied with wanting to know if that software does what our customers want once it gets released. The Lean Startup movement has shown us the importance of putting our software in front of our customers, then working out how they actually use it so we can determine what to do next. In this world, I was struck by the shortcomings of the tools in this space. Commonly used web analytics tools, for example, might only help me understand tomorrow how my customers used my site today.

Read more…

Comment
Four short links: 18 February 2014

Four short links: 18 February 2014

Offensive Security, Sage-Quitting, Ethics Risks, and War Stories

  1. Offensive Computer Security — 2014 class notes, lectures, etc. from FSU. All CC-licensed.
  2. Twitter I Love You But You’re Bringing Me Down (Quinn Norton) — The net doesn’t make social problems. It amplifies them until they can’t be ignored. And many other words of wisdom. When you eruditely stop using a service, that’s called sage-quitting.
  3. Inside Google’s Mysterious Ethics Board (Forbes) — nails the three risk to Google’s AI ethics board: (a) compliance-focus, (b) internally-staffed, and (c) only for show.
  4. 10 Things We Forgot to Monitor — devops war stories explaining ten things that bitly now monitors.
Comment

Computing Twitter Influence, Part 1: Arriving at a Base Metric

The subtle variables affecting a base metric

This post introduces a series that explores the problem of approximating a Twitter account’s influence. With the ubiquity of social media and its effects on everything from how we shop to how we vote at the polls, it’s critical that we be able to employ reasonably accurate and well-understood measurements for approximating influence from social media signals.

Unlike social networks such as LinkedIn and Facebook in which connections between entities are symmetric and typically correspond to a real world connection, Twitter’s underlying data model is fundamentally predicated upon asymmetric following relationships. Another way of thinking about a following relationship is to consider that it’s little more than a subscription to a feed about some content of interest. In other words, when you follow another Twitter user, you are expressing interest in that other user and are opting-in to whatever content it would like to place in your home timeline. As such, Twitter’s underlying network structure can be interpreted as an interest graph and mined for insights about the relative popularity of one user when compared to another.
Read more…

Comment

The True Cost of Lemonade

Learn to resist vanity metrics

One of the things we preach in Lean Analytics is that entrepreneurs should avoid vanity metrics—numbers that make you feel good, but ultimately, don’t change your behavior. Vanity metrics (such as “total visitors”) tend to go “up and to the right” but don’t tell you much about how you’re doing.

Many people find solace in graphs that go up and to the right. The metric “Total number of people who have visited my restaurant” will always increase; but on its own it doesn’t tell you anything about the health of the business. It’s just head-in-the-sand comforting.

A good metric is often a comparative rate or ratio. Consider what happens when you put the word “per” before or after a metric. “Restaurant visitors per day” is vastly more meaningful. Time is the universal denominator, since the universe moves inexorably forwards. But there are plenty of other good ratios. For example, “revenue per restaurant visitor” matters a lot, since it tells you what each diner contributes.

What’s an active user, anyway?

For many businesses, the go-to metric revolves around “active users.” In a mobile app or software-as-a-service business, only some percentage of people are actively engaged. In a media site, only some percentage uses the site each day. And in a loyalty-focused e-commerce company, only some buyers are active.

This is true of more traditional businesses, too. Only a percentage of citizens are actively engaged in local government; only a certain number of employees are using the Intranet; only a percentage of coffee shop patrons return daily.

Unfortunately, saying “measure active users” begs the question: What’s active, anyway?

To figure this out, you need to look at your business model. Not your business plan, which is a hypothetical projection of how you’ll fare, but your business model. If you’re running a lemonade stand, your business model likely has a few key assumptions:

  • The cost of lemonade;
  • The amount of foot traffic past your stand;
  • The percent of passers-by who will buy from you;
  • The price they are willing to pay.

Our Lean lemonade stand would then set about testing and improving each metric, running experiments to find the best street corner, or determine the optimal price.

Lemonade stands are wonderfully simple, so your business may have many other assumptions, but it is essential that you quantify them and state them so you can then focus on improving them, one by one, until your business model and reality align. In a restaurant, for example, these assumptions might be, “we will have at least 50 diners a day” or “diners will spend on average $20 a meal.”

The activity you want changes

We believe most new companies and products go through five distinct stages of growth:

  • Empathy, where you figure out what problem you’re solving and what solution people want;
  • Stickiness, where you measure how many people adopt your solution rather than trying it and leaving;
  • Virality, where you maximize word-of-mouth and references;
  • Revenue, where you pour some part of your revenues back into paid acquisition or advertising;
  • Scale, where you grow the business through automation, delegation, and process.

Read more…

Comment
Four short links: 11 April 2013

Four short links: 11 April 2013

Automating NES Games, Code Review Tool, SaaS KPIs, and No Free Lunch

  1. A General Technique for Automating NES Gamessoftware that learns how to play NES games and plays them automatically, using an aesthetically pleasing technique. With video, research paper, and code.
  2. rietveld — open source tool like Mondrian, Google’s code review tool. Developed by Guido van Rossum, who developed Mondrian. Still being actively developed. (via Nelson Minar)
  3. KPI Dashboard for Early-Stage SaaS Startups — as Google Docs sheet. Nice.
  4. Life Without Sleep — interesting critique of Provigil as performance-enhancing drug for information workers. It is very difficult to design a stimulant that offers focus without tunnelling – that is, without losing the ability to relate well to one’s wider environment and therefore make socially nuanced decisions. Irritability and impatience grate on team dynamics and social skills, but such nuances are usually missed in drug studies, where they are usually treated as unreliable self-reported data. These problems were largely ignored in the early enthusiasm for drug-based ways to reduce sleep. […] Volunteers on the stimulant modafinil omitted these feedback requests, instead providing brusque, non-question instructions, such as: ‘Exit West at the roundabout, then turn left at the park.’ Their dialogues were shorter and they produced less accurate maps than control volunteers. What is more, modafinil causes an overestimation of one’s own performance: those individuals on modafinil not only performed worse, but were less likely to notice that they did. (via Dave Pell)
Comment