"testing" entries

Test-driven Infrastructure with Chef

Velocity 2013 Speaker Series

If you’re a System Administrator, you’re likely all too familiar with the 2:35am PagerDuty alert. “When you roll out testing on your infrastructure,” says Seth Vargo, “the number of alerts drastically decreases because you can build tests right into your Chef cookbooks.” We sat down to discuss his upcoming talk at Velocity, which promises to deliver many more restful nights for SysAdmins.

Key highlights from our discussion include:

  • There are not currently any standards regarding testing with Chef.  [Discussed at 1:09]
  • A recommended workflow that starts with unit testing  [Discussed at 2:11]
  • Moving cookbooks through a “pipeline” of testing with Test Kitchen [Discussed at 3:11]
  • In the event that something bad does make it into production, you can roll back actual infrastructure changes. [Discussed at 4:54]
  • Automating testing and cookbook uploads with Jenkins [Discussed at 5:40]

You can watch the full interview here:

 

How to Get Fast, Cheap Feedback on Your Product with Tiny Tests

User research you can do now

There’s a lot of advice about how to do great user research. I have some pretty strong opinions about it myself.

But, as with exercise, the best kind of research is the kind that you actually DO.

So, in the interests of getting some good feedback from your users right now, I have some suggestions for Tiny Tests. These are types of research that you could do right this second with very little preparation on your part.

bottom-micrometer

What is a Tiny Test?

Tiny Tests do not take a lot of time. They don’t take a lot of money. All they take is a commitment to learning something from your users today.

Pick a Tiny Test that applies to your product and get out and run one right now. Oh, ok. You can wait until you finish the post.

Unmoderated tests

Dozens of companies now exist that allow you to run an unmoderated test in a few minutes. I’ve used UserTesting.com many times and gotten some great results really quickly. I’ve also heard good things about Loop11 and several others, so feel free to pick the one that you like best.

Read more…

Four short links: 10 December 2012

Four short links: 10 December 2012

Regular Expressions, Mobile Diversions, UX Pitfalls, and DIY Keyboarding

  1. RE2: A Principled Approach to Regular Expressions — a regular expression engine without backtracking, so without the potential for exponential pathological runtimes.
  2. Mobile is Entertainment (Luke Wroblewski) — 79% of mobile app time is spent on fun, even as desktop web use is declining.
  3. Five UX Research Pitfalls (Elaine Wherry) — I live this every day: Sometimes someone will propose an idea that doesn’t seem to make sense. While your initial reaction may be to be defensive or to point out the flaws in the proposed A/B study, you should consider that your buddy is responding to something outside your view and that you don’t have all of the data.
  4. Building a Keyboard: Part 1 (Jesse Vincent) — and Part 2 and general musings on the topic of keyboards. Jesse built his own. Yeah, he’s that badass.

Tools for test-driven development in Scala

Two core Scala libraries support features for mocking and data generation.

Scala, a language designed for well-structured and readable programs, is richly provisioned with testing frameworks. The community has adopted test-driven development (TDD) and behavior-driven development (BDD) with zeal. These represent the baseline for trustworthy code development today.

TDD and BDD expand beyond the traditional model of incorporating a test phase into the development process. Most programmers know that ad hoc debugging is not sufficient and that they need to run tests on isolated functions (unit testing) to make sure that a change doesn’t break anything (regression testing). But testing libraries available for Scala, in supporting TDD and BDD, encourage developers to write tests before they even write the code being tested.

Tests can be expressed in human-readable text reminiscent of natural language (although you can’t stretch the comparison too far) so that you are documenting what you want your code to do while expressing the test that ensures that code ultimately will meet your requirements.

Daniel Hinojosa, author of Testing in Scala, describes the frameworks and their use for testing, TDD, and BDD in this interview.

Highlights from our discussion include: Read more…

The many sides to shipping a great software project

An interview with Shipping Greatness author Chris Vander Mey.

Chris Vander Mey, CEO of Scaled Recognition, and author of a new O’Reilly book, Shipping Greatness, lays out in this video some of the deep lessons he learned during his years working on some very high-impact and high-priority projects at Google and Amazon.

Chris takes a very expansive view of project management, stressing the crucial decisions and attitudes that leaders need to take at every stage from the team’s initial mission statement through the design, coding, and testing to the ultimate launch. By merging technical, organizational, and cultural issues, he unravels some of the magic that makes projects successful.

Read more…

Data Jujitsu: The art of turning data into product

Smart data scientists can make big problems small.

Having worked in academia, government and industry, I’ve had a unique opportunity to build products in each sector. Much of this product development has been around building data products. Just as methods for general product development have steadily improved, so have the ideas for developing data products. Thanks to large investments in the general area of data science, many major innovations (e.g., Hadoop, Voldemort, Cassandra, HBase, Pig, Hive, etc.) have made data products easier to build. Nonetheless, data products are unique in that they are often extremely difficult, and seemingly intractable for small teams with limited funds. Yet, they get solved every day.

How? Are the people who solve them superhuman data scientists who can come up with better ideas in five minutes than most people can in a lifetime? Are they magicians of applied math who can cobble together millions of lines of code for high-performance machine learning in a few hours? No. Many of them are incredibly smart, but meeting big problems head-on usually isn’t the winning approach. There’s a method to solving data problems that avoids the big, heavyweight solution, and instead, concentrates building something quickly and iterating. Smart data scientists don’t just solve big, hard problems; they also have an instinct for making big problems small.

We call this Data Jujitsu: the art of using multiple data elements in clever ways to solve iterative problems that, when combined, solve a data problem that might otherwise be intractable. It’s related to Wikipedia’s definition of the ancient martial art of jujitsu: “the art or technique of manipulating the opponent’s force against himself rather than confronting it with one’s own force.”

How do we apply this idea to data? What is a data problem’s “weight,” and how do we use that weight against itself? These are the questions that we’ll work through in the subsequent sections.

Read more…

In defense of frivolities and open-ended experiments

Our children will improve upon the things we're building in ways we can't conceive.

Before you scoff at the pointlessness of yet another social network, web app, or project, remember that we don't always do the research or build the company that is immediately useful or profitable.

Jason Huggins' Angry Birds-playing Selenium robot

How a game-playing robot could help shape the future of mobile testing.

If you try to talk to Jason Huggins about Selenium, he'll probably do to you what he did to us. He'll bring his Arduino-based Angry Birds-playing testing robot to your interview and then he'll relate his invention to the larger problems of mobile application testing and cloud-based testing infrastructure.

Jason Huggins’ Angry Birds-playing Selenium robot

How a game-playing robot could help shape the future of mobile testing.

If you try to talk to Jason Huggins about Selenium, he'll probably do to you what he did to us. He'll bring his Arduino-based Angry Birds-playing testing robot to your interview and then he'll relate his invention to the larger problems of mobile application testing and cloud-based testing infrastructure.

Don’t put all your trust in mobile emulators

Steve Souders on how he reduces the development risks of mobile emulators.

Steve Souders, performance evangelist at Google, looks forward to the remote capabilities of debugging and testing, but he warns against putting too much faith in emulators.