It's not exponential, it's sigmoidal

Over Thanksgiving dinner, Saul Griffith was complaining about the lack of mathematical literacy among people who should know better. “Take all that talk about the exponential growth of various web sites. Don’t people realize that those curves are actually sigmoidal?”

exponential vs. linear and cubic curves sigmoidal curve
Exponential vs. linear or quadratic curves. A sigmoidal curve.

And of course, he’s right. These curves look exponential but eventually they do flatten out. In fact, one of the most important sigmoidal functions is the logistic function, originally developed to model the growth of populations. Wikipedia notes: “The initial stage of growth is approximately exponential; then, as saturation begins, the growth slows, and at maturity, growth stops.” In fact, most of these curves aren’t even sigmoidal, they are sinusoidal. (This is, incidentally, why Ray Kurzweil is most likely wrong about the singularity.)

So when you look at Alexa, and see something like this:

3 year facebook graph from alexa

Remember that over a longer timeframe, it will come to look more like this:
5 year myspace graph from alexa

tags:
  • Ben Greenberg

    Great article. It gives some much needed perspective on all kinds of trends and fads, not just internet ones.

    I’m not convinced about the singularity either, but one of the main points in Kurzweil’s book is that technological progress does proceed in a sigmoidal fashion.

    I can’t remember if he used the term, but he talks about exponential growth that does eventually flatten out, but then resumes exponential growth after another technological breakthrough. And if you look at a sigmoidal graph on a large enough timescale, it looks exponential.

  • Alex Tolley

    It is perfectly acceptable to call any curve exponential if that is a first order description of its trajectory. The issue is not the description, but the understanding that nothing exponentiates to infinity. Since it is very difficult to determine the upper bound of a sigmoidal curve, it is pointless trying to introduce this description of a growth curve.

    As a further note, there is nothing that indicate that the eventually trajectory should be sigmoidal or sinusoidal. A simple example is speculative price peaks in financial markets.

  • I have been trying with very little success to explain this point as it relates to the power law.

  • Kurzweil is hopelessly optimistic, but he isn’t just about the exponentials. He actually has a model of a series of sigmoidal bursts. He calls them S-curves though (he is pop-science after all).

    From that article you link:

    We also need to distinguish between the “S” curve (an “S” stretched to the right, comprising very slow, virtually unnoticeable growth–followed by very rapid growth–followed by a flattening out as the process approaches an asymptote) that is characteristic of any specific technological paradigm and the continuing exponential growth that is characteristic of the ongoing evolutionary process of technology. Specific paradigms, such as Moore’s Law, do ultimately reach levels at which exponential growth is no longer feasible. Thus Moore’s Law is an S curve. But the growth of computation is an ongoing exponential (at least until we “saturate” the Universe with the intelligence of our human-machine civilization, but that will not be a limit in this coming century). In accordance with the law of accelerating returns, paradigm shift, also called innovation, turns the S curve of any specific paradigm into a continuing exponential. A new paradigm (e.g., three-dimensional circuits) takes over when the old paradigm approaches its natural limit. This has already happened at least four times in the history of computation. This difference also distinguishes the tool making of non-human species, in which the mastery of a tool-making (or using) skill by each animal is characterized by an abruptly ending S shaped learning curve, versus human-created technology, which has followed an exponential pattern of growth and acceleration since its inception.

  • Minor correction to Ben Greenberg’s comment: While any particular technology will approximate a sigmoidal curve, multiple technologies overlapping in time produce an exponential curve for the underlying measure, whether that is storage, computation, or bandwidth.

    Right now we seem to be approaching a architectural discontinuity WRT computation in that additional cycles are likely to be added by increasing parallelism rather than faster clocks, but the underlying cycle/$ measure continues to march apace with no sign of letting up.

  • Ivan and Michael —

    My disagreement with Kurzweil is not over whether successive overlapping sigmoidal bursts even out to an exponential, but whether even that kind of cycle repeats indefinitely.

    Why does Ray think that modern civilization is indifferent to history. I’m sure that in the second century AD, things looked pretty good to the Romans.

    We assume that the progress of technological civilization is a given. But it wouldn’t take much to wreck the whole thing. You have only to consider the rise of fundamentalism in America, with its anti-science agenda, throw in a couple of environmental and economic collapses, and a few wars, to see the whole thing come to a screeching halt.

  • Tim, you’re certainly right. Continued progress is neither certain nor particularly likely. Perhaps the situation is fundamentally different once an artificial general intelligence starts improving itself. But even in the optimistic scenario of decades of development till we reach that point, progress could be killed by many things.

  • This is a great article, technologies have limits, systems have limits, even if they are currently growing almost exponentially (there was a great spoof of the old Cisco ads “Soon there will be more people on the Internet than there are on Earth”).

    Tim: I would add “for a while” to the end of your last sentence. Robert Wright’s book Nonzero offers some interesting arguments in this regard.

    “We assume that the progress of technological civilization is a given. But it wouldn’t take much to wreck the whole thing. You have only to consider the rise of fundamentalism in America, with its anti-science agenda, throw in a couple of environmental and economic collapses, and a few wars, to see the whole thing come to a screeching halt.”

    As De Tocqueville observed “democracies believe in the infinite perfectability of man.” There may very well be limits to what we can achieve, but certainly we can travel beyond the limits of the current horizon. I don’t know that I believe in the “Singularity” either but you can look at a number of significant projects (e.g. Human Genome) where the completion time collapsed exponentially.

  • Jonathan

    I agree with Alex Tolley. I’d go further to say that it is counterproductive to provide this sigmoidal argment. It has been shown that people reason very badly about the implications of nonlinear growth as it is, and this will just confuse them further.

  • The ‘S’ shaped curve (or the sigmoidal curve) is well recognized in the diffusion of innovation theory, for example market penetration of a technology. However, with the passage of time, a disruptive technology starts another S shaped curve that eventually crosses the existing curve.

  • Andrew Hamilton

    In marketing, this is known as the Bass diffusion model: en.wikipedia.org/wiki/Bass_diffusion_model.

  • Geoff B

    I agree.

    A mature technology that lives on the outer section of an S curve is usually 1) a bit of a disappointment, or 2) ready for a serious disruption.

    Let’s hope the efficiency of the internal combustion engine falls in the latter category.

  • Thanks for reminding us of such sobering realities

    I guess any growth will go sigmoidal once saturation reaches near 100% – take windows market penetration. The important point is to keep in mind that saturation for a given product or service may be something _very_different_ from 100%.

    Thanks for using mySpace and Facebook as examples ;-)

  • David Mercer

    Assuming nothing lasts forever, the curves aren’t even sigmoidal; they’re bell curves.

  • Dennis

    Kurzweil: what’s different now, compared to history, is that we’ve got computers. As Moore’s Law continues (and it’s got room to keep going for decades), computational power drives progress in other technologies.

    Technology is all he’s predicting, though. War, drought, peak oil, economic crash, these sorts of things are beyond his analysis. But historically, they haven’t hindered technology all that much.

  • as with most modern theorizing, science discovered this a long time ago.

    epidemiology is the study of this sort of behavior. mathematical models of epidemics has been around for a long time.

    also, the analogy between social networks and epidemics is just too perfect.

    a nice graph and simple summary is at: http://mvhs1.mbhs.edu/mvhsproj/epidemic/epidemic.html

    joshua

  • It’s important to distinguish the level of technical acumen of one’s audience–otherwise you risk being accurate at the cost seemingly being pedantic.

    Technically, a sigmoid IS also an exponential function, but most folks simply aren’t numerate enough to understand the difference.

    I also routinely substitute the term “geometric” for exponential in populist lectures and literature. It helps since most folks can relate to “doubles” growths from the rice-chessboard parable, compound interest from financial instruments, or common measurements like decibels or the Richter scale.

    The reverse is also true. One time an HR person tried to explain the results of a company survey to our research group. She got essentially ripped to shreds as we pointed out the errors in the curve-fitting and regression analysis done on the survey data–in real-time.

    I later sent her a quick apologetic note saying that she must’ve drawn the “short-straw” assignment. She was tasked with spinning a new corporate policy based on conclusions which weren’t supported by the survey data–to the data mining & performance research groups. Heh.

  • anonymous

    Quiet Tim, you’ll confuse the investors. ;)

  • geeknerd

    Assume that Kurzweil’s static analysis of arbitrary ‘data points’ is valid. A quick look at a plot of energy usage (something approximating real, measurable data) reveals his wishful thinking. Oh wait, that’s right, Technology Jesus, I mean the Singularity, will allow exponential increase of energy usage for an indefinite period. Anyone care to wager that something depending on input of limited resources follows a logistic curve?

  • bayareaguy

    “Why does Ray think that modern civilization is indifferent to history. I’m sure that in the second century AD, things looked pretty good to the Romans.”

    Having recently read the story of Carasius on Wikipedia, I’d say that many must have thought otherwise.

    See http://en.wikipedia.org/wiki/Carausius

    Also, the link at the bottom makes for a very interesting read as well http://www.romanbritain.freeserve.co.uk/CARAUSIUS.HTM

  • steve

    The sigmoidal result is an exponential operating with restrained resources. When the result is also solving the resource problem, then the exponential continues (until the new resource limit).

    That is why Kurzweil is almost certainly correct about the Singularity. The fundamental constants of human nature are being changed by technology, with the emergence of networked group minds.

    Unfortunately, you’ll either get this or you won’t. A random comment on a blog isn’t going to change your mind.

  • Hmmm —

    I’m not sure what Carausius has to do with this, other than that by the end of the third century, the party was definitely over. That’s why I chose the second century in my comment, during the Antonine period, where the expansion was over, but everything seemed to be perfect and would go on forever as it was.

    I agree that this historical anecdote isn’t just about technological progress. But technological advance depends on certain social conditions.

  • steve —

    The idea that the result is solving the resource problem is unproven.

    While I believe that there is something new emerging with networked group minds (and in fact, in many circles, I’m considered the high priest of at least one branch of that cult, with all my writings about web 2.0), I don’t think that changes the fact that there are many externalities that have the potential to change the march of technological progress.

    You could argue that the rise of city states and the modern concept of the individual was as profound a change as that which is happening today — and in fact, was perhaps the last major turn of that spiral — but it didn’t lead to unbroken progress either.

    Relative to hunter gatherer existence, that was a huge leap. (See Guns, Germs and Steel for a nice summary of the argument.)

    And in many ways, Rome was the pinnacle of that stage (with another one in China.) And it was a pinnacle that was lost, in a sinusoidal curve that took us a long way down. When Gibbon wrote the Decline and Fall in the late 18th century, many Roman achievements had yet to be recreated.

    So while it’s possible that we’ll manage to achieve some kind of escape velocity, it’s far from a foregone conclusion.

  • steve

    Well argued. While I think that a minority of the human race will achieve some form of escape velocity (by entering space), I acknowledge the counter-points you have raised.

    There’s also the consideration that the form of life that continues on that path may not be what the rest of us would call humanity. For example, I’d (jokingly) point to teenagers and how their constant use of SMSs has altered the form of data processing to wide but shallow.

    I am particularly interested in brain plasticity and the way it is possible to connect electrode plates to the back of the tongue and “see” through it… what if that became some form of ECG with feedback?

    Then what if there were a group that had lived that way for years, then you extracted a single member – would the result be a person, or a frament of something?

  • I think it amazing that anyone truly can believe that growth can ever truly be exponential, I think you are right with anything successful there are always limits to the growth, usually based on the physical limitations and interest of those engaging with them, for example, even though it always seems there is a new blogger around the corner, there actually is a limit to how many blogs will be started and maintained as the number dropping off eventually over-whelms those starting.

    Such is the nature of all new systems, to spur new periods of rapid growth there needs to be some sort of catalyst, and when the catalyst has been spent the curve begins to slow.

  • Greetings Tim,
    I agree that Ray’s vision is a little bit too distorted, but in this case, technology doesn’t seem to have any “saturation” cap that can transform its evolution from exponential to sigmoidal/synusoidal… unless we take into account the probability that something very terrible happens in the next year (nanowars? a new virus? who knows?).

    Apart from that, the general idea of sigmoidal growth is very right (my 2 cents).

  • Doesn’t this just simply say, that the computer industry is doomed to economic laws like every other sector (product) is. Is this the discovery of “saturation”?

    But I agree that it’s really cool to say a sentence like “…that those curves are actually sigmoidal?”

    I am sure, there’s stunning all around, I’m going to use it on my next conference speech :-)

  • Simone —

    That’s the point: there are ALWAYS external factors that come into play.

    The summer of 1914 was one of the most beautiful in living memory, and everyone thought that we were entering a period of unparalleled peace and prosperity.

    Now a critic will say, yes, but WWI didn’t end technological progress, and I agree with that. It’s a rare historical event that truly ends the progress of technology.

    But it’s happened before. It happened with Rome, it happened with Arab civilization. China didn’t fall, but it stagnated. And all of the other great civilizations of the world eventually came to some kind of catastrophic end.

    There are so many things that could derail the progress towards the “singularity.”

    One I adverted to in my post: the rise of religious fundamentalism. Living in a world that embraces technology and progress, it’s hard for us to believe that much of our government is dominated by people who don’t, who in fact embrace millenial visions of the end of the world in our lifetime.

    When our vice-president is a keynote speaker at a conference organized by Tim LaHaye, that’s really scary.

    The point isn’t that any particular thing is going to happen, just that the future doesn’t go conveniently in the direction we expect.

    A really good example is space travel. Would you have expected, after the moon landing in 1969, that we’d be unable even to go back 35 years later? Not only have we lost the capability to go to the moon, I remember talking to some employees of Jeff Bezos’ Blue Origin, and he said that the fundamental engine technology hasn’t changed since then.

    Technologies fall “out of fashion.” We also hit limits. For example, imagine the scenario where we hit the end of Moore’s law at about the same time as there’s a biotech revolution, and all the smart money says, OK, let’s go over here instead, and leaves computer technology on an evolutionary pinnacle that it won’t get beyond for another thousand years.

    (Everyone forgets just how interrelated everything is. Computer chips don’t just get better automatically. Someone has to decide to make a multi-billion investment in a new fab. At some point, those billions may bring better returns elsewhere. At which point, the fundamental driver of Moore’s law goes away.)

    I’m not saying that this *will* happen. I’m saying it might happen. And that any study of exponentials will tell us that it’s only in theory that they go on forever.

  • Rainer,

    Yes, the idea that there’s a saturation point is the essence of sigmoidal curves.

    I remember also something that Frank Herbert wrote in Dune as “the Law of the Minimum,” namely that growth is limited by the key nutrient that is present in the least amount.

    Or I think of the philosophy of Karl Jaspers (Way to Wisdom is a good place to start), and his argument that understanding the limits of our existence is the source of both wisdom and happiness.

    I’m all for the rise of AI. We might even get there. But I don’t think that even true AIs will be as disruptive as Ray thinks they will be.

  • alex tolley

    “But I don’t think that even true AIs will be as disruptive as Ray thinks they will be.”

    Tim, if true AIs eventually emerge, why do you think they won’t be very disruptive? Everything I know suggests otherwise.

  • Good point. This needs repeated saying. I tried to make similar points at the Singularity Summit, but
    it’s a crowd with a singular focus. Back in 1999 I tried to curb the irrational exuberance of those times with Norvig’s Law: “Any technology that surpasses 50% penetration will never double again (in any number of months)”.

  • alex — I didn’t say they wouldn’t be disruptive, just “not as disruptive as Ray thinks they will be.”

    Every major new technology is disruptive. Look how the printing press, steam power, railroads, automobiles, airplanes, radio and television, the personal computer, and the internet have changed the face of our civilization. Genomics is on the verge of doing so.

    But that’s a far cry from “technological change so rapid and profound it represents a rupture in the fabric of human history.”

    Such changes ARE human history. My guess is that rather than a singularity, we face a future in which what seems extraordinary to us today simply comes to seem normal. I don’t remember who said it, but “Technology is anything that was invented after you were born,” seems apposite.

    Those who talk about the singularity are just showing their age.

  • Machine will never surpass human intellect (technological singularity). Machines have helped us create new technology that, in turn, has increased human intellect. Machines will continuously help enhance human intellect.

    The rupture in the fabric of human history occurs with any major advancement of technology, be it the printing press or the web.

    The accelerated change in technology that Ray talks about will happen when nano, bio, and information technology will merge together to help us create technology (and diffuse through out the mass) 100 times faster than what is possible today. Still, the technology and machines that we will create will not surpass human intellect because with the help of those machines we would leap forward in our own intellect.

    I love Crichton’s Prey.

  • Tim Gray

    “My disagreement with Kurzweil is not over whether successive overlapping sigmoidal bursts even out to an exponential, but whether even that kind of cycle repeats indefinitely.”

    Well, they don’t have to repeat indefinitely, just long enough…

  • Tim Gray —

    True enough. But there’s a decent chance that “long enough” is long enough for the probability to go way down below the level he articulates.

  • > This is, incidentally, why Ray Kurzweil is most
    > likely wrong about the singularity.)

    Actually, Kurzweil refers to continued break-through followed by plateau.

    … Not one s-curve but continued s-curves which appear to continue as a double exponential curve.

    Paul Reilly