Sun

May 8
2005

Tim O'Reilly

Tim O'Reilly

Tech: A hostile environment for US Programmers?

On Dave Farber's IP list, Jonathan Shapiro makes some insightful comments on software outsourcing, and the inevitability of the economic trends driving outsourcing. He says, "The right question isn't 'How do we keep programmers employed?' The right question is: 'How do we change the economics of software?'"

Shapiro ends up sounding many themes you've heard from me before -- how open source and free software teach us that the value isn't in the software anyway, and how as a result we need to move up the stack to new sources of value. ("Invest in the free software process, since that price cannot be undercut. Shift our business attention to other parts of the value chain.") He also points out that if we want to keep our lead in innovation, we need to fix the software patent system. But that's not all. He suggests rethinking the way we train programmers, with a focus on engineering discipline and programmer productivity.

He also suggests, counter-intutively, removing liability protection for software vendors. "No better way to employ programmers than to finally force us to rebuild everything correctly --and for that matter, to discover *how* to rebuild things correctly." Eloquent and thoughtul. Very well worth a read.

tags:   | comments: 14   | Sphere It
submit:

 
Previous  |  Next

0 TrackBacks

TrackBack URL for this entry: http://blogs.oreilly.com/cgi-bin/mt/mt-t.cgi/4026

Comments: 14

  Chad Fowler [05.08.05 10:36 AM]

These sentiments are remarkably similar to those expressed in my upcoming book. While predators such as Lou Dobbs feed on the fears of Americans, urging them to blame their companies, their government, and their offshore competitors---anyone but themselves---as an industry we're missing the point. There is no "bad guy". Economic forces are changing the face of the software industry. As programmers, if we continue to fail to hang on to our traditional jobs, we have no one to blame but ourselves.

Jonathan Shapiro is right to doubt the viability of tariffs and other government intervention as a "long-haul" solution. His suggestions are powerful and relevant. However, I find it disturbing that his solutions center on wide sweeping political and economic change rather than change from the bottom up.

His message implies that the changes that need to take place are not things that each of us can make personally. The only individual course of *action* proposed is the formation of unions while we lobby against the practices of our employers. As he said, it's in the employers' best interest to hire at the lowest price. Ultimately, is it not also in the *economy's* best interest?

If we're in reinvention mode, why not start with ourselves? "Success" in the software profession can't continue to be defined as it has been in the past. Shapiro's message loses consistency at the end. It's as if he's urging programmers to hang on for dear life while the economy sorts itself out. It's this kind of lazy, "Someone else will do it for me" attitude that has put livelihoods at risk.

We need to take control of our careers. And, that doesn't mean fighting over pennies and trying to win our old jobs back. If the "value isn't in the software", then fighting for your old programmer job is a sure-to-be-bloody battle over an increasingly insigificant prize.

  steve h. [05.09.05 02:38 PM]

------


Lou Dobbs feed on the fears of Americans, urging
them to blame their companies, their government,
and their offshore competitors---anyone but
themselves---as an industry we're missing the
point. There is no "bad guy".


------


Uh, I'd say there's always plenty of bad guys when there are strong motivators like money involved.
As far as programmers blaming everybody but themselves, not sure I really see why they should blame themselves. And what I don't see from you is anything concrete about what sort of thing they can do for themselves. Except maybe "get out of programming"? Is that it? For many of us, programming is so much a part of who we are that we will do it after work hours if that's the only way we can. Just a shame that it has to come to that. And please don't start it with "reinvent yourself as a programmer" stuff, many of us have always done that anyway, but the price pressure from cheap labor markets is impossible to compete against given the cost of living here in the US.
If that's not a legitimate complaint, I don't know what is.

  Bob [05.09.05 06:03 PM]

Lou Dobbs a predator? I guess O'Reilly is a prophet.

When does it become apparent to you that social responsibility is important? How many people went to school to learn programming with the promise of a better life? In addition, started to build on that promise only to see it disappear after a very short time. You do not know the nature of business yet. You seem too young. There are many companies out there like Enron.

When the time comes when you become irrelevant, then I guess it will become a crisis. I will wait.

  Jeroen Wenting [05.09.05 11:54 PM]

"invest in the open source process" sounds nice but why invest in that using US (or European if you're a European company) people and not hire some Indians or Romanians for the job?

The question should indeed be how we keep our own people employed and proficient. If we don't, sooner rather than later we'll find the roles of the 18th century reversed with Europe and north America becoming the backwards nations where the only available jobs are unskilled labour and truck driver (replacing agriculture which is also being offshored to countries with cheap dirt and labour) and we'll be relying on the leftover crumbs of the Asian power economies.

  Tim O'Reilly [05.10.05 07:36 AM]

Jeroen, "invest in the open source process" doesn't mean hiring people to work on open source projects (although that might be a small part of the investment). It means two things to me:


  1. Rethinking the role of open source in your company's software -- to use Geoffrey Moore's language, understanding what is "context" and what is "core". Leverage open source as context so you can invest in your own unique core capabilities. Look at the analogy in computer hardware. While Compaq diddled around trying to improve the commodity PC hardware platform with hardware innovations, Dell cleaned their clock by seeking out the lowest cost hardware anywhere on the planet, and innovating in business processes. IBM completely gets this now: if there's open source software they can use, they suck it in, and add value where other people haven't already built what is now part of the computer industry's fertile commons.


  2. Using lessons from open source to understand better how to build software in the internet era, with distributed teams, community involvement, users as testers, etc. (This is the core business of Collab.Net.) This is one example of the "process innovation" corresponding to point 1.

When you think about open source in this way, outsourcing (which is better thought of as globalization) is no more threatening than open source itself. Yes, it threatens established players and established ways of doing things, but it also opens vast opportunities for the creation of new value.

  Kevin Farnham [05.10.05 08:13 PM]

Much of my 12/29/2004 blog entry (www.MathematicalAnalysis.com/blog) addresses this issue. Some of the main themes:

There is no reason to think the late 1990s will re-appear. History and economics don’t work that way. There was a bubble because expectations were at odds with reality.

1970s - 1990s software development is a job for which the market pricing is in decline. Why? Because this type of development is no longer incredibly new and innovative, which means that a much larger portion of the world’s populace is capable of performing the tasks, which means that if we have free and open markets and inexpensive transport (the internet’s rather inexpensive, I’d say) the work will go to the locales where the tasks can be performed cheaply from the point of view of corporations, but where the salaries are desirable from the point of view of the populace that is capable of performing the work tasks well.

This is nothing new. It is the history of the United States to invent, to expand and develop (providing great jobs for U.S. citizens), then to see those jobs be “exported” to cheaper producers after the technology ceases to be new.

What must happen in order for the situation to become once again excellent for software developers? Innovation! Just as we’ve always done in the past, we must innovate, create the next world-changing technology.

Is there any way we can somehow work to “jumpstart” the redeployment of financial resources here in the U.S., toward innovation growing out of the ideas of our own highly skilled technology workforce? Is there any way that we could stimulate a sprinkling of “seed” money?

Ideas are needed. There may not be lots of software development jobs available, and jobs of the old type will continue to disappear. But, there are jobs that have not yet been invented, because the ideas have not yet been presented to those who might be willing to invest in an innovation that is not much more than an idea at present.

  Jeroen Wenting [05.11.05 01:07 AM]

Tim, of course open source isn't just paying people to write free software.
But if there's no people being paid to write software noone's going to write free software either...

So if the programming profession disappears from north America and Europe (as is happening) then soon the only way to get any programming done is to go to Asia (for example) and that's where all the open source stuff will be too.
Open source sounds nice but it rarely can be used without modification to suit specific demands (a million frameworks and generic libraries but nothing built on them is a typical situation in the OS landscape) so people will have to be paid still except there will be no people left to do the work for you (at least not at home)...

In the meantime the economies of your own home market will have crashed to the point where there's noone left to buy your products or services (as all potential customers are unemployed...) and you'll find yourself with a rather poor cashflow unless you move the company to India and try to beat the established companies there to regain some market share.

  Mr. Joad [05.11.05 01:43 AM]

The problem is that most of us can't afford to 'reinvent' ourselves, whatever that might mean. There are two roads to reinvention: 1) An entreprenural route (which involves great financial risk for most people) and 2) the education route (going back to school - this is very expensive as well and while you're going back to school you tend to lose a huge chunk of income in addition to paying for school)



Even one of the biggest cheerleaders for globalisation, Thomas Friedman (author of "The Earth is Flat" ) admits that it's going to be very tough for US workers given the attitude of the current administration which seems to be more intent on investing in weaponry than in people. Friedman (heard him on Charlie Rose the other night) says that in light of the rapid pace of globalisation Washington should be changing direction in a big way by investing in education at all levels (from kindergarten to PhDs), but instead we're bent on spending $Billions to fight a war predicated on false pretenses.



Reinvention is necessary, but it requires great risk. Government could play a role in helping to mitigate some of that risk for the average citizen. That could be done by helping people (many of whom are in their 30's, 40's, 50's and have mortgages to pay and kids to raise) more with education expenses and providing seed capital to start new businesses.



But instead we're passing bankruptcy reform (all slanted towards creditors benefit) which makes taking risks even more... risky. Instead we're cutting education budgets in many areas. Instead DARPA is changing from fianancing a lot of pure research (the kind of research which led to the internet itself) to financing research which has an immediate, short-term pay-off on the battlefield.



...And after all that 'reinvention' it may well be that you still won't be able to find steady work. We might have to face the fact that our turn is up and now it's time for other nations which are thinking further ahead and which are not encumbered by massive debt to take center stage. You can do a lot of things on a personal level, but global forces can render those personal actions meaningless. The best course of action at this point might be to reduce our expenses and eliminate debt in order to ready ourselves for a lower standard of living in the US. We've probably got to be ready to live on a much lower income (30 to 50% less) in order to compete on a global scale. As a culture, we've become fat and spoiled, addicted to amusements and consumption - it's been a long time since we've been 'hungry'.



As for Lou Dobb's being a 'predator': Who is he preying on? He's just sounding the alarm. The news isn't very good, but why shoot the messenger?

  Kevin Farnham [05.11.05 08:40 AM]

Blaming the current administration for what's happening is silly. Do you then hail the previous administration for presiding over a situation where software develop jobs were "created" (temporarily) based on ridiculous illusory expectations about the future that resulted in companies that had yet to make a quarterly profit having valuations higher than many established giants on the Dow 65 list?

Global economics and the tides of history are relentless. The power of any presidency is minuscule in comparison.

I completely disagree that we (Americans) did anything wrong, that it's because "we've become fat and spoiled" that software jobs are leaving the U.S. The reason is pure economics: the cost of transport of the goods has gone to near zero (from the internet), and a large supply of well-educated workers who live in regions where the cost of living is much lower is readily available. Therefore, the jobs will leave the U.S.

But does this mean the U.S. economy and standard of living are doomed? Not necessarily, since we are still inventing new technologies, as we always have. As a whole, the nation's economy should move ahead. But that doesn't mean our displaced software engineers won't be left behind.

If the new jobs are only in other areas, such as biotechnology, where will displaced software engineers find work? That's why I think the only hope for software engineers is new software ideas and new start-ups. But venture capital money simply has not flowed since the 2000 bust. That, to me, is the most damning news of all.

In the early 1990s a famous and scary book entitled "Decline and Fall of the American Programmer" was quite popular. The author (Edward Yourdon) later "recanted" in subsequent books, after the rise of the internet put even more American programmers to work. For this reason, I try to stay positive. But, on the other hand, I know it's different this time: rise of the internet has forever leveled the global playing field for software development and other services. Yourdon may simply have espoused an accurate prophesy a decade too soon.

  Mr. Joad [05.11.05 11:52 AM]

Blaming the current administration for what's happening is silly


The current administration probably isn't to blame for the current situation, however they could certainly be responding to the current situation in a more pro-active way.


Global economics and the tides of history are relentless. The power of any presidency is minuscule in comparison.
I totally agree, but the current administration could at least recognize that large segments of the population are being negatively impacted and at least propose some potential remedies that might help the US economy make the transition (the edcucation funding and seed money for starting businesses as I proposed above).


I completely disagree that we (Americans) did anything wrong, that it's because "we've become fat and spoiled" that software jobs are leaving the U.S.
I guess I wasn't clear with that part of my comments. Look at history, it happens over and over again. There are cultures/societies/nations that rise to prominent positions in the world (Rome, Brittain, Spain) and stay there for a time but then fall and it can often be traced to complacency. We (in the US) could be experiencing that same kind of decline here. It's a pattern that always repeats: rise, wealth accumulation, military pre-eminance and then eventually a complacency sets in. I'm not saying we did anything wrong, per se, it's just a natural cycle.


But does this mean the U.S. economy and standard of living are doomed? Not necessarily, since we are still inventing new technologies, as we always have.


Perhaps, but this almost sounds like the nearly religious belief that Americans are more innovative than citizens of other countries. Why would that be so? Can't other nations become innovative? Definately we're seeing that happen now. The US doesn't have a lock on innovation. And as more and more engineering jobs move to other places, more innovation will be happening outside of the US.


That's why I think the only hope for software engineers is new software ideas and new start-ups. But venture capital money simply has not flowed since the 2000 bust. That, to me, is the most damning news of all.


Exactly. The governemnt could play a role by providing seed money for small startups. We're not talking about huge sums of money here. You don't need much capital to start a software-startup. However, you do need money to live on as you get things going. It can take a few years for a startup to become profitable. Why not offer a couple of years of living expenses for people who are willing to take the risk? Two years max at something like $25K to $30K/year (not a lavish salary by any measure, but a person could easily live on it).

  Kevin Farnham [05.11.05 01:27 PM]

Mr. Joad -- reading your clarifications, I think we agree much more than we disagree.

I really like your last point, about a government program that provides funding for micro-start-ups. The start-ups that succeeded (even though the success percent might be low) would easily pay back the cost of the program by paying corporate taxes and providing excellent jobs for people (bringing them off the unemployment list and back into a taxpaying role).

  Murr.... [06.17.06 09:23 PM]

Humm....

It's fathers day, the local papers leading article is the number of foreclosures in the state. It notes the policy of gov'mnt and the behavour of indviduals - says nothing about the fundamental changes in the economy.

Money is like heat, it flows from hot to cold. Flowing out of CA to CO in the 80's was good for CO, Sun moved, SGI did not - draw your own conclusions on the grid. But now ... the USA money is flowing else where...

I used to work construction. The union companies were superior in the quality of work and the level of managment - and the happiness of the workers. Union contractors had to pull from the same labor pools at the same labor rate with the same ratio of newbies to journeymen. The only way for a company to underbid the other company was to manage the job better. Did you understand? The pressure was on managment to do well vs who was going to make the men work for the lowest dollar. Read that last sentence again. And by the way, the union compaines made more money for thier owners becuse the dollars were bigger the margins were bigger. However;

The UNION workers coveted thier "shade under the tree". Even when there was hughe amounts of work the labor unions keept the labor pool of elibles low, letting "thier boys" work large hours of overtime vs adding more workers to "the international brotherhood". And then it happend, someone started a construction company with all the out of union ( not out of work ) men and the industry has been spiriling downward ever since. Spiriled all the way down to mexico.

So... few state born people want to work in construction and even fewer will encourage thier children to work in that business. I'm not going to recommend IT to my children as a career choice.....


What then shall we do?

  br [08.16.06 03:05 AM]

Not everyone can program a computer. yet, despite all the hard work, its becoming a cheap labor job. Not worth it anymore, unless you want to move to India.

  Jonthan Shapiro [10.07.06 10:04 AM]

I only just found this thread.

In response to Chad Fowler's comments: I share his suspicions of sweeping change. But there is a real problem here: people follow the incentives imposed by their environment. Developers are no exception.

While many developers are doing a lot of things from the bottom up to improve things, many more are not, and the ones who do try operate in an environment where these improvements are marginalized. The software world runs in a mostly "open loop" fashion -- there is no feedback that incentivizes software producers to invest in quality or correctness. In the absence of this incentive, the question for the vendor is: "can I afford to incur an extra cost that won't really help me sell product but will make my development costs higher than those of my competitor?" The answer, clearly, is "no". In those companies that fail to recognize this, shareholders can be trusted to quickly remind them. To make matters worse, the flaws in software are so pervasive, and the reporting of flaws so limited, that it is very difficult for customers to see any visible quality advantage between two vendors. Customers will not pay for higher quality that they cannot see.

Think about it: Diebold released a bunch of voting machines whose software was laughably vulnerable into a market sector where people actively cared about vulnerability. When they got caught (which shouldn't have surprised *anybody*) there response was to threaten legal action against the watchdogs claiming that the software had been stolen, and throw a whole bunch of FUD and confusion into the press, all the while denying the problem. The state of the practice is "deny, deny, threaten, deny."

So: the problem is not merely that there is no *positive* reinforcement of quality and engineering in the absence of liability. The problem is that in a competative environment there is *negative* reinforcement.

On the other hand, if the environment shifts in a way that rewards discipline, talent, and expertise, the trend toward software as a blue-collar job will stop overnight.

The problem at that point will be that the American computer science education system doesn't look to be up to the job of producing the programmers we will need...

Post A Comment:

 (please be patient, comments may take awhile to post)






Type the characters you see in the picture above.

RECENT COMMENTS