Previous  |  Next

Fri

Jan 4
2008

Tim O'Reilly

Tim O'Reilly

Human vs. Machine: The Great Challenge of Our Time

In an email conversation with Bill Janeway about our upcoming Money:Tech Conference, he said something really profound:

The timeliness of this Conference is NOT only because "web 2.0" technologies and business models have reached critical mass in the financial markets. It is also because, as driven by the web more generally, the frontier between human and machine-decision making has become radically problematic. First, quantitative approaches in trading, pricing, valuation, asset definition vastly expanded the domain for machine decision-making. But then the humans struck back, by refusing to act like the mindless molecules that the models driving machine decison-making required. The self-reflective, behavioral attributes of human market participants is now driving back that frontier, requiring innovations in every aspect of financial market processes, beginning with techniques of risk measurement and risk management. When price is an inverse function of liquidity and liquidity is an inverse function of price certainty, the recursive loop can only be broken by human intervention and action.

I've written quite a bit about "bionic software," the idea that one of the distinguishing characteristics of Web 2.0 is that its applications are a new hybrid of man and machine, driven by algorithmic interpretation of aggregated human activity. Recent turmoil in financial markets show us just how such systems can run amok.

Figuring out the right balance of man and machine is one of the great challenges of our time. We're increasingly building complex systems that involve both, but in what proportion?

Bill Janeway will be talking at the conference with Rick Bookstaber, author of A Demon of Our Own Design. Bookstaber was the head of risk management for Morgan Stanley, and now runs a hedge fund. He argues that the very techniques originally developed to manage risk via computational means have actually increased risk. He asks whether we can put the genie back in the bottle, and whether we can afford not to.

Incidentally, this same issue is playing itself out in the world of Web 2.0 itself, with new search engines, from Jason Calacanis' mahalo to Jimmy Wales' Wikia Search making the argument that a purely algorithmic approach is fundamentally flawed. In response to yesterday's announcement of Wikia Search, Cory Doctorow wrote, in a BoingBoing editorial entitled Wiki-inspired "transparent" search engine:

We have a notion that the traditional search engine algorithm is "neutral" -- that it lacks an editorial bias and simply works to fulfill some mathematical destiny, embodying some Platonic ideal of "relevance." Compare this to an "inorganic" paid search result of the sort that Altavista used to sell.

But ranking algorithms are editorial: they embody the biases, hopes, beliefs and hypotheses of the programmers who write and design them.

Mahalo is placing a bet on human intervention in search results; Wikia Search on the power of making its ranking algorithms open and transparent (a la open source software). But both are trying to re-draw the boundary between human and machine.

For what it's worth, while Google strongly favors a proprietary algorithmic approach (much like hedge funds and Wall Street firms trading for their own account), they also recognize the importance of human intervention. Peter Norvig, formerly the Director of Search Quality at Google and now its Director of Research, pointed out to me that there is a small percentage of Google pages that dramatically demonstrate human intervention by the search quality team. As it turns out, a search for "O'Reilly" produces one of those special pages. Driven by PageRank and other algorithms, my company, O'Reilly Media, used to occupy most of the top spots, with a few for Bill O'Reilly, the conservative pundit. It took human intervention to get O'Reilly Auto Parts, a Fortune-500 company, onto the first page of search results. There's a special split-screen format for cases like this.

One also sees human vs. machine in the battle of search engines such as Google against search-engine spam. When I pinged him about the subject of this post, Peter wrote to me:

I do think you have a good point -- as more money is handled by automated trading systems rather than by human traders, there is a larger risk of chaotic behavior. You are right that there is an analogy to search result manipulation, but I think we search engines actually have an easier problem because we can control the timescale and magnitude at which we make changes, whereas in markets big changes can happen very fast, and the only brake is to close the market...

It's also intriguing to see how humans start to adapt to algorithmic models, learning their deficiencies and gaming the system. Search engine spam is a great case in point. Radar reader Eric Blossom noted recently, in a comment on my post Trading for their own account:

I find myself skipping to the second page or so of the search results list that I get from Google. I do this to avoid the heavy commercial pages that seem less pertinent. Places like Digg also seem to collect pointers to blogs etc. rather than to original sources. I'm clicking more as my eyeballs spin than I used to. Perhaps the golden age of search engines is already over from the user's perspective. They're still useful. Just a bit more painful.

But of course, if more and more people start to act like Eric, that could help to inform Google's algorithms that some of those second page results are actually to be preferred, leading the system to "heal" itself. This would be a positive feedback effect from human reaction to algorithmic misbehavior. But it could go the other way.

One also has to wonder if there could, in the future, be a catastrophic sub-prime-like crash for the Google adwords market. There are more dissimilarities than there are similarities. An Adword is not a derivative. But as I suggested in my original Release 2.0 issue on Web 2.0 and Wall Street, all it would take is a desperate second-tier search engine introducing futures pricing into keyword buys (versus today's real-time auction) to securitize and potentially destabilize this market.

George Soros' comment from the introduction to his book The Open Society is relevant here:

I interpret history as a reflexive process in which the participants' biased decisions interact with a reality that is beyond their comprehension. The interaction can be self-reinforcing or self-correcting. A self-reinforcing process cannot go on forever without running into limits set by reality, but it can go on long enough and far enough to bring about substantial changes in the real world. If and when it becomes unsustainable, it may then set into motion a self-reinforcing process going in the opposite direction. Such boom-bust sequences are clearly observable in financial markets, but their extent, duration, and actual course remain uncertain.

It's probably less valuable to speculate about specific ways that the Web 2.0 economy could go haywire than it is to recognize that financial markets, as large, networked information systems driven by a combination of algorithmic and human activity, may well be canaries in the coal mine for other similar systems (like Web 2.0 applications) as they continue to evolve.

Remember William Gibson's dictum: "The future is here. It's just not evenly distributed yet."

tags: algorithms, bionicsw, money:tech, Web_2.0  | comments: 18   | Sphere It
submit:

 

0 TrackBacks

TrackBack URL for this entry: http://orm3.managed.sonic.net/mt/mt-tb.cgi/2768

Isaac Mao   [01.04.08 06:16 AM]

I ever talked about a new model about hybrid computing human plus machine. The best case can be used on social search and social portfolio(not social graph the buggy term). We can test such a hybrid computing model with google+twitter to see some really relevant result about some issues like "China","Web 2.0", etc. The social layer of Twitter has dramatically improved the quality of search result.

Alex Tolley   [01.04.08 06:51 AM]

"The self-reflective, behavioral attributes of human market participants is now driving back that frontier, requiring innovations in every aspect of financial market processes, beginning with techniques of risk measurement and risk management. When price is an inverse function of liquidity and liquidity is an inverse function of price certainty, the recursive loop can only be broken by human intervention and action."

To err is human, to really f**k up requires a computer? :-)

What evidence has Janeway got for his statement? The logic of using machines to drive trades and business on Wall Street is inescapable. The greatest current debacle - sub-prime loans - was caused by humans, not machines. The argument than only humans can break recursive loops (or self referential ones) is absurd and an echo of the disbelief in AI.

nüchtern   [01.04.08 06:56 AM]

if you know what youre looking for and have alot of web-affinity you can easily read the search engine result pages and quickly know what is for you and what is not. it might be a few more clicks but thats what tabs are made for isnt it? :)

Galerie   [01.04.08 08:22 AM]

Unfortunately it is not so easey to create a human with a machine inside :o))

chris   [01.04.08 09:14 AM]

What you write about the dramatic intervention required to get a Fortune 500 company onto the first page of results for its name will come as no surprise to those who understand why disembodied AI does not and certainly never will work. Among many other things, intelligence is about dealing with novel things and edge cases. AI cannot do that.

The Berkeley professor Hubert Dreyfuss is the guy who first called AI out on this in the 1960s while he was at MIT. In his Fall 2007 lecture on Heidegger "Critique of Descartes II" (from 29 minutes in) he gives an amusing account of how that worked out for him. Every serious researcher in AI now believes that his kind of criticisms were right, even if they don't buy into all the German philosophy he derived it from.

@Alex Tolley: Even AI researchers don't believe in disembodied AI anymore. If even Google - a very clever company that is obsessed with getting this kind of stuff right and employs lots of clever AI researchers like Peter Norvig - needs humans to get a Fortune 500 company onto the front page of its search results, then that suggests that there are no AI systems out there that can function in a dynamic environment as we would like them to without human intervention. And how can disbelief in AI be absurd when even Marvin Minsky is on record as having said, "AI has been braindead since the 1970s"? Surely a continued belief in the kind AI that might drive trades without occasionally catastrophically screwing up is what is absurd.

Thomas Lord   [01.04.08 11:30 AM]

To underscore Chris a bit:

"disembodied intelligence" is an oxymoron. (e.g., start with rudy rucker and get eventually to "Where Mathematics Comes From")

A problem is that classic AI techniques are a magicians trade -- you can't get to the "I" in "AI" with those classic techniques but you sure can dazzle people. This is a problem.


-t

Michael R. Bernstein   [01.04.08 02:32 PM]

"It's also intriguing to see how humans start to adapt to algorithmic models, learning their deficiencies and gaming the system."

BTW, I'd consider a machine able to game the system to be a good milestone along the way to strong AI.

It's worth noting that as economics increasingly relies on simulations to find the ways in which humans do *not* act as collections of rational agents (simple example: vengeance in iterated tit-for-tat), it also comes closer to being able to simulate a human economic agent.

Willi Schroll   [01.04.08 03:23 PM]

Human vs. Machine: The Great Challenge of Our Time


The title hits the point



The quote of George Soros: "I interpret history as a reflexive process in which the participants' biased decisions interact with a reality that is beyond their comprehension." is philosophically deep and different thinkers through the centuries have made that point. Most prominently the rise and fall of marxist-leninist "history control" have shown that Marx was wrong in his hybris and Hegel et al. were right with their humility.



In the 21st century the new temptation of hybris is the "total machine" - the ubiquituous dependency of human activity from the internet, i.e. machine activity.



As an analyst of the ongoing explosive web evolution I am neutral, but as a thinking citizen I strongly welcome technologies with participative character - the (social) web2.



At the crossroads today (Human vs. Machine) we always should vote for human control and participation, freedom and responsibility.

Tim O'Reilly   [01.04.08 06:11 PM]

Alex --

Bill Janeway's reply (via blackberry email from India):

"Well, I certainly do "disbelieve" (where did that word come from?) in AI, along the lines argued by the Dreyfus Brothers in "Mind over Machine". No question that human beings have long made dumb loans that were never going to be repaid to busted borrowers in order to collect fees...but they could not package them into CDOs and model their value based on historical correlation data without benefit of actual trades until they had machines to help"

Alex Tolley   [01.04.08 07:45 PM]

Straw man alert. I certainly did not say anything about "disembodied AI". However, since AI is clearly a contentious and distracting example, let me rephrase my argument.

Janeway stated:
1. P = 1/f(L), where L = 1/f(P).
2. Only humans have the capacity to break out of this "recursive" loop.

Machines can handle this self-reference and work effectively. In 1987, simple portfolio insurance models exacerbated the 1987 stock market crash, because the same models mindlessly enacted the same recommendations. But game theory could have been applied to those models to provide some meta-modeling of what happens when other actors are doing the same thing. Easily programmable, just wasn't.

Janeway's larger point is that machines are fallible. No surprises there, they are programmed by humans. But rather than blaming humans for their fallibility of the system they have collectively created, he prefers to argue that the silly machines were too dumb to operate in a more complex environment. I think it is equally plausible to postulate an arms race between humans trying to arbitrage parts of the system and the programmers and their machines to chase these opportunities. Haven't we seen exactly that approach with machine learning and rules based expert systems in determining credit card fraud?

It should be clear that I do think that humans and machines have an almost symbiotic relationship in this regard. However, I do not accept the thesis that machines "only do what they are programmed to do" as if they were mindless, automatons. They can be programmed to offer creative solutions, at least within narrow domains, as work by Doug Hofstadter and others have shown. Doug Lenat even did a Google presentation on Cyc to show how it could aid in search.

Search◊ Engines Web   [01.04.08 10:31 PM]

For what it's worth, while Google strongly favors a proprietary algorithmic approach (much like hedge funds and Wall Street firms trading for their own account), they also recognize the importance of human intervention. Peter Norvig, formerly the Director of Search Quality at Google and now its Director of Research, pointed out to me that there is a small percentage of Google pages that dramatically demonstrate human intervention by the search quality team. As it turns out, a search for "O'Reilly" produces one of those special pages. Driven by PageRank and other algorithms, my company, O'Reilly Media, used to occupy most of the top spots, with a few for Bill O'Reilly, the conservative pundit. It took human intervention to get O'Reilly Auto Parts, a Fortune-500 company, onto the first page of search results. There's a special split-screen format for cases like this.
One reason for this inequity is the motivation for other top sites to voluntarily give you back links as a constantly updating information site. Commercial sites can attempt aggressive SEO to compensate, but Google is against some SEO tactics link buying links with high PR or paid blog or directory links.

Hence, the deepening organic inequity.

Here is a related post from Google's spam chief in December ...

http://www.mattcutts.com/blog/subdomains-and-subdirectories/

For several years Google has used something called “host crowding,” which means that Google will show up to two results from each hostname/subdomain of a domain name. That approach works very well to show 1-2 results from a subdomain, but we did hear complaints that for some types of searches (e.g. esoteric or long-tail searches), Google could return a search page with lots of results all from one domain. In the last few weeks we changed our algorithms to make that less likely to happen in the future.

This change doesn’t apply across the board; if a particular domain is really relevant, we may still return several results from that domain. For example, with a search query like [ibm] the user probably likes/wants to see several results from ibm.com. Note that this is a pretty subtle change, and it doesn’t affect a majority of our queries. In fact, this change has been live for a couple weeks or so now and no one noticed. :) The only reason I talked about the subject at PubCon at all was because someone asked for my advice on subdomains vs. subdirectories.

__________________________________________

On another note, could you please share your blog traffic stats for 2007, including screenshots of:
Top referrers
Top Keywords
Top Posts
etc

Many top bloggers are doing it, so the concern with hiding them is slowly disappearing in the openness of the Web 2.0 era.

Thomas Lord   [01.04.08 10:50 PM]

(I said "rudy rucker" when, obviously, I actually meant rodney brooks. I've long been quite bad with proper names, for some reason.)

-t

chris   [01.05.08 04:32 AM]

@Alex Tolley

You didn't use the word "disembodied", but Cyc is the ultimate example of disembodied AI, that is to say, the attempt to reproduce AI in a static computer. No straw men. Embodied AI, by contrast, is the kind of intelligence you'll find in a robot tasked with, say, vacuum-cleaning floors, although my floor-vacuuming robot is still unintelligent enough to keep getting trapped under the sofa. If disembodied AI is AI, then embodied AI is AI 2.0.

Notions of embodiment and disembodiment are pretty standard in modern AI. Here's a good quote from the BBC from Sept 2001 that is quite relevant:

"AI used to be seen as recreating disembodied rational intelligence," said Dr Inman Harvey, an evolutionary roboticist from the School of Cognitive and Computing Science at the University of Sussex, UK. "We're getting away from disembodied intelligence and turning to the adaptive behaviour of simple, though still very complex, animals."

A Google search on > "artificial intelligence" disembodied

BTW, Cyc is an abject failure and an embarrassment to the AI community. Its ability to help with Google searches is not evidence of the intelligence of Google's algorithms or of Cyc's database. The Wikipedia entry on Cyc quotes from a recent book that Cyc is "one of the most controversial endeavours of the artificial intelligence history," and I'd suggest that is a charitable description. Everything that is wrong with good-old-fashioned AI is exemplified by Cyc.

Web 2.0 is about extending and (mainly) distributing cognition. It is not about AI. AI and cognition-extending or -distributing tools are completely different things. If Web 2.0 were about AI it would have failed by now - or have moved into robotics.

Tim O'Reilly   [01.05.08 09:54 AM]

Alex,

I don't want to argue for Bill's position by proxy. Nor do I want to argue that in a future of even more intelligent algorithms, the need for human intervention may become less and less, but the point is that NOW, we are trying to find the right balance.

As various operations of significance are increasingly run by algorithms, and as humans both game those algorithms, and act unexpectedly for reasons not programmed into them, humans still need to intervene.

In Web 2.0, we're pushing on both frontiers: how to use algorithms more intelligently, and how to have humans intervene more intelligently. E.g. closing the market till the humans (and the machines) get their brains back is a bit like a reboot on a PC: OK, we're not sure what kind of a mess we're in, but let's stop everything and start over. It's easier than figuring out what's going wrong.

But what wikia and mahalo are trying to do with their search engines is to bring humans in in a different way.

There are lots of strategies for human involvement. Some are pre-algorithmic (e.g. pagerank) and some are post-algorithmic (e.g. mahalo). Some aggregate humans statistically, some provide mechanisms for explicit human interaction. Some are controlled and private, others (e.g. wikipedia) open and promiscuous.

But all modern "intelligent" systems are in some sense a fusion of human and machine, and adjusting the balance between the two, and finding new ways to combine the two, is where we will have new breakthroughs, and new opportunities.

In some sense, every Web 2.0 success story is a story about how some human figured out how to build a machine that harnesses the activity of other humans...

Alex Tolley   [01.05.08 06:20 PM]

After saying that AI was a distraction I have to mention Cyc, setting Chris off again. Doh!
But I will provide one final tweak:
"Machine learning is a subfield of artificial intelligence (AI)...". from "Programming Collective Intelligence", p3. Toby Segaran, O'Reilly 2007.

Tim,
I don't disagree with anything in your last comment. I strongly agree with your characterization of the fusion of human and machine capabilities in intelligent systems.

Where I would nuance your comments is that most software algorithms are used to compute deterministically, such as derivatives prices. However there is a lot of scope to work in the non-deterministic arena too, and that is where we may see some interesting ideas emerge. You have often talked about data being the next "Intel inside", and I see Web 2.0 apps using novel data collections that depend on human agency for their creation. I suggest that software that can capture some of the human creative cognitive processes to work on this data will be a fruitful area of innovation. In the financial context, software that can "recognize" market maker liquidity changes from trade data on prices, size and participants might be used to warn of a market meltdown and suggest exchange closure.

BillyWarhol   [01.05.08 08:05 PM]

It is my Hope that a whole New Worldwide Internet Economy will be created from the foundations of Web2.0 + the Blogging Community! The Best part is it will eliminate Poverty + Hunger + Provide Clean Drinking Water + Food + Shelter + Clothing + Medicine to the Poor People who Need it NOW!!

Hopefully the Money Conference will be addressing that too* If no one is I will do it* We think our WEB3D.0 Initiative is going to Change the World for Better*

;))

Peace*

Michael R. Bernstein   [01.05.08 08:54 PM]

BTW, it's worth noting that search is one of those areas that SF got very wrong. There was a strong assumption as late as y2k that search-like functions would be dominated by decentralized mobile agents (devised, 'bred', or some combination by individual humans) operating in a very local context, essentially browsing the internet by proxy and pre-chewing their take for their human masters. We can still see shades of this show up occasionally in movies.

Now, it is possible that the relatively decentralized approach could see a renaissance, and if so, it would likely be via digital signatures and DRM to allow 3rd-party code to run on centralized services, or alternatively via having a local copy of the digested index on your desktop (or some hybrid approach), but right now, the freedom of the network belongs to he who owns one (especially botnets).

Barak Hachamov   [01.07.08 08:07 AM]

The 1st Human Operating System - Facebook /
From DOS to HOS
http://www.blonde2dot0.com/blog/2007/12/21/the-1st-human-operating-system/


Post A Comment:

 (please be patient, comments may take awhile to post)




Remember Me?


Subscribe to this Site

Radar RSS feed

RELEASE 2.0 BACK ISSUES

BUSINESS INTELLIGENCE

CURRENT CONFERENCES