To paraphrase Irving Kristol, “an indexer is stock-picker who has been mugged by reality.” We all begin our investing careers assuming that with the right amount of intelligence, education, knowledge and experience we can consistently beat the market. That seductive belief, enhanced by a natural tendency toward overconfidence, blinds us to the fatal flaw in our reasoning: the assumption that the market is rational.

The first thing an indexer has to learn is that the market doesn’t need to make sense. As Garfield Drew pointed out, “Stocks don’t sell for what they are worth but for what people think they are worth.” The dot-com and real estate bubbles of the last decade are only two recent examples that people don’t always think rationally.

Nevertheless, human nature demands a rational explanation for everything, even if that explanation involves more magical thinking than common sense. Bubbles—be they real estate, stocks or even Beanie Babies—require justification, so good reasons for their continued existence are invented and accepted.

The more sophisticated investor embraces Drew’s observation and takes the next logical step, which John Maynard Keynes compared to picking the winner of a beauty contest. Keynes wrote that the goal is not to pick the most beautiful contestant but to choose the contestant the average judge will consider the most beautiful. These sophisticated investors try to determine what average investors are thinking and beat them to the punch.

Beyond Sophisticated

If that isn’t tough enough, Keynes went on to posit that there are ultra-sophisticated investors who take it one step further, by trying to anticipate the actions of the sophisticates. And beyond that, he continued, may be another iteration of supremely sophisticated investors attempting to predict what the ultras will think and so on and so on. It can make your head spin.

This is the attraction of technical analysis, particularly during market bubbles. Technical analysis is more concerned with what traders are willing to pay for stock certificates than it is with a company’s economic value.

But even knowing the right strategy for this market and the right strategy for the next market isn’t enough. You also have to get the timing right. You have to know when to change strategy.

And if this doesn’t convince you that indexing makes more sense than picking stocks, add in Donald Rumsfeld’s “unknown unknowns”—the things we don’t know we don’t know. There is always the risk of the inconceivable “black swan” destroying an otherwise brilliant investment strategy.

Despite the compelling logic, it is extremely difficult to completely abandon the idea that we can pick the winners or time the market. It took me 25 years before deciding to index and yet, 10 years later, I find myself struggling with the cognitive dissonance of thinking I can accurately forecast the future while knowing I cannot. Human nature may be controlled, but it can’t be eliminated.

Kent Grealish is a partner at Quacera Capital Management, a fee-only advisory firm. He welcomes comments and suggestions for future columns at: kent@quacera.comThis e-mail address is being protected from spambots. You need JavaScript enabled to view it .