Static analysis with C#

Complement a good testing program and identify hard-to-find bugs with static analysis.

Static analysis is, quite simply, any analysis you perform on software without actually running it. (Analyzing software as it runs is dynamic analysis.) There are many reasons to do static analysis, but almost all of them boil down to the desire to improve software quality. As a designer of developer tools, improving software quality by any means is keenly important to me.

Let’s consider compiler warnings. They are produced without executing the code, so the compiler is doing static analysis. Their aim is to inform the developer that the code, though legal, is probably wrong. Suppose you were a compiler developer and you wanted to add a new warning; what characteristics must that warning have?

  • There must be some statically identifiable pattern to the suspicious code.
  • The pattern must be common and plausibly written by a developer; developing a warning for a too-rare pattern or completely unrealistic code is effort that could be better spent on other features.
  • The warning must have a low “false positive” rate; a warning must actually identify defective code more than, say, 99% of the time. False positives encourage developers to eliminate the warning by turning the warning off, or worse, by incorrectly changing the code. There must be a way to eliminate the warning without introducing a bug into the code.
  • The pattern must be identified extremely Slowing the build process by anything more than a few percent is unacceptable.

I always recommend that everyone use the strictest warning settings on their compiler, to pay attention to warnings, and to (carefully) fix them all. Even fix the false positives; if the code was weird enough to fool the compiler then it’s weird enough to fool a human, and you don’t want to have “expected” warnings distracting you from actual warnings.

Compiler developers have good reasons to make warnings only about common, easily detected, easily fixed defects that require only shallow, fast analysis to attain a >1% false positive rate. That is a very small fraction of possible defects!

Now think about warnings from the point of view of a product manager trying to get software out to customers on budget. Experienced product managers know that the costs of fixing a serious, customer-impacting defect go up enormously over time. A defect caught on the developer’s machine by a compiler warning might slow the developer down by only a few minutes. If it gets caught by automated tests after the developer goes home then that’s significantly more time and effort wasted the next day. Catching the defect with manual testing adds even more costs. And as we saw many times in 2014, the cost of major defects like heartbleed and Apple’s “goto fail” defect is enormous and affects the whole economy.

A product manager doesn’t have the same constraints as a compiler developer; their goal is not to make a compiler that runs fast, but to ship a product that does not crash customer machines or open security holes in customer networks. Many customer-impacting defects have characteristics opposite of those that make a good compiler warning:

  • The defective pattern could show up only once in a million lines of code. Most code is correct, and some critical defects are exceedingly rare.
  • Relatively high false positive rates are common; algorithms which identify, say, potential SQL injection attacks often have false positive rates in excess of 10%.
  • Identifying the defective pattern (and eliminating false positives) could require whole-program analysis that examines millions of data flow and control flow scenarios. Compilers must analyze millions of lines of code per minute; whole-program static analyzers can get away with millionsper hour.
  • The project might be using a language like JavaScript that doesn’t even have a “compile” step!

There are now several static analysis tools available that can take a deep look at your source code and find mission-critical customer-impacting bugs long before customers do; Apple’s “goto fail” defect would have been easily found by a good static analyzer.

If deep static analysis is cost-decreasing, world-saving goodness, why do I get either blank stares or negative reactions when I ask people about their use of analysis tools? I can think of several reasons.

Many developers are simply unfamiliar with the concept of standalone code analysis tools, and you can’t use a tool that you don’t know exists. These tools are in developer consciousness today the same way that source control and bug databases were decades ago; barely at all.

Worse, developers who have experience with static analysis tools often perceive them as being more trouble than they are worth. Analysis tools with high false positive rates, or tools which only identify trivial “fit and finish” issues can bias developers against higher quality tools.

A common argument I hear is “we don’t need static analysis because we have good testing“. There is no substitute for good testing, it’s true. But it is a rare test bed indeed that can exercise more than a tiny fraction of all the control flows and data flows through a program. Concurrent programs are particularly difficult to test; deadlocks and memory corruptions may happen so rarely that testing does not find them, but the vastly larger set of customer’s machines will. (See below for an example.)

Another common argument is “we don’t need static analysis because we have a strong culture of code reviews“. Again, there is no substitute for developers critiquing their colleague’s code. But a defect that escaped one developer will likely escape others as well; a concurrency defect like this C# defect (adapted from a real open-source program) can easily get past even experienced developers:

It can be hard to see the defect even knowing that there is a defect present, and having the key portions of it highlighted. (Many threads can pass the initial check, they queue up to enter the lock, and then all threads will increment the current position, potentially moving it past the end position, contrary to the intention of the check.) But a static analyzer looking for the pattern “a variable used in a condition tested outside of a lock might change inside the lock” can find this defect.

Static analysis is not a substitute for either good testing or smart developers, any more than source control or bug databases are; rather, all those things make humans more productive by enlisting computers to do the tasks that humans are bad at.

Finally, static analysis tools themselves need to continue to improve in order to more clearly demonstrate their enormous benefits. Many static analysis tools do not yet integrate well into modern “all-in-the-cloud” workflows, do not analyze the full range of languages used in typical codebases, and so on.

Fortunately, signs are pointing towards a tipping point for analysis tools of all sorts. Microsoft’s recent open-sourcing of the “Roslyn” engine for creating community-authored C# and Visual Basic analyzers, the increasing usage of libraries such as clang for analyzing C, C++ and Objective C, and other similar tools, are all welcome advancements. I am very much looking forward to the day, coming soon, when advanced analysis tools are as much a part of the air we breathe as our code repositories and bug databases.

tags: , , , ,