# AnalyticBridge

A Data Science Central Community

The stock market is much more random than what most investors think. Indeed, it looks almost like a random walk: even if it has been going down 7 days in a row, the probability that it will go dow tomorrow is still 50%: random walks are memory-less processes. The chart below illustrates a simulated, perfectly random / unpredictable stock market.

• Why can't you consistently win? Events that can boost or kill the stock markets are quickly exploited by professional traders. This explains the gap between yesterday close price, and today open price. Indeed many traders don't keep open position overnight because of this.
• For large indices (QQQQ etc.), the correlation with today's price and the price from yesterday or from k days ago (k = 1, 2, 3, etc.) is essentially zero. If it was significantly different from zero (from a statistical point of view), then this pattern would have been exploited by professional traders - many holding a MIT or Princeton PhD in mathematics - and eventually the pattern would die. Note that the time series correlogram (some call it the spectral signature) associated with stock prices uniquely identifies the nature of the underlying stochastic process. Time series with a flat correlogram represent pure random walks, that is, processes where you can not predict the future.
• So is the stock market truly random? No. It is true that when a large number of independent algorithms (run by hedge funds) compete in real time to extract money from Wall Street, eventually nobody wins or lose - and the stock market becomes as random as a lottery: some can win for some time, but on the long term they can't. The return becomes zero. Think of your daily commute when you are stuck on highways driving at 5 miles an hour: can you find a pattern to beat the rat race, maybe a new road that you can use to beat your fellow commuters? If you do, that road will be found by thousands of commuters fighting the same battle to minimize commute time, and your miracle road will turn into rat race again, in a matter of days. The same mathematical principles (arbitrage) apply to the stock market. However, even if you exclude insider trading and after hours trading, the market, while almost random, is not entirely random. Most of Wall Street trading algorithms have been designed by mathematicians trained in the same universities, and are not independent: it is like all search engine algorithms providing the same search results for most keyword searches. This indeed explains why the stock market can experience flash crashes or flash spikes.
• This discussion about randomness applies to short-term trading. But is there a long-term trend that can be leveraged? In my opinion, since year 2000, the stock market has no up or down trends: while highly volatile, it is mostly flat. I think there's a good chance for a slow downward trend given the fact that baby boomers are retiring, demographic pressures, and the young generation having lost trust in 401K plans. Yet government might impose some regulations to artificially keep the market from collapsing - otherwise long term shorting would be a great option for the smart people who are currently "all cash".
• Does it mean that there's no money to be made for the average investor? Well, it is becoming more like a lottery, and a few people win money, sometimes big money, in all lotteries. Look at my above chart showing a purely random (simulated) stock market: there are local trends, patterns such as short squeeze, "head and shoulder", market crash, steady growth, runs (going up or down for 7 days in a row). Every kind of pattern can be found in this simulated, random, trend-less stock market, just like in the real stock market. It is clear that you could make money in this simulated market - what is less clear is that you have no way to predict if your algorithm will or will not make money. Note that to increase your odds, you have to back test your strategies and perform sound cross-validation or "walk-forward" (http://en.wikipedia.org/wiki/Walk_forward_optimization) to predict what your return might be.
• Despite the increasingly random nature of the stock market, it is still possible, for the average educated trader, to exploit patterns. Typically, any pattern that Wall Street is reluctant to exploit - such as staying all cash for 5 years in a row and trading like crazy when the right opportunity presents itself. In a nutshell, Wall Street traders must think very short term or otherwise risk losing their job. This provides an opportunity for the amateur trader. As far a short-term patterns are concerned, we've found that they occur no more than three times, and can be exploited only during the third occurrence (the first two occurrences being used for pattern detection and confirmation): after 3 occurences, Wall Street gurus have exploited and killed the pattern (the patten - e.g. a short squeeze on a particular stock - might still be there, but it's parameters have shifted on the fourth occurence, due to heavy exploitation).
• The future could be rosier: with more traders leaving this extraordinary competitive environment (e.g. they leave because they are laid off due to poor performance) and less money circulating in the stock market due to baby boomers retiring, the nature of the stock market will become less random, whether the general market trend is up, down or neutral. This will provide new opportunities for traders who haven't been killed in the 2000-2010 "wild fire".

Views: 14226

### Replies to This Discussion

Mathematics explains patterns.

Fundamentals explain systems.

Lyapunov exponents in this system are probably positive , where the time constant are getting shorter and shorter the instability will get getting and greater. Chaotic behavior and mode competition is something I would expect to see more often especially with automated trading. There comes a point when all these systems oscillate between each other as competing hedge fund super computers try and out manipulate each other...

Its not really random it just looks that way in the linear world. If someone has the time use Wigner Voltera time series reconstruction and you would probably be able to recover he chaotic dynamics underneath the noise. Its all non linear, dont waste your time on linear analysis if you want to understand it, your variances will be too high to make sense of it all.

is your analysis for stock only, or do you include derivatives (options) ?

stock only

Wouldn't it make sense that the future price behavior of stocks and options would be mathematically tied in some kind of relationship - given Black/Scholes, et. al.

Therefore, whatever analysis would apply to stocks would also apply to options.

Also, obviously, as our times become more and more turbulent (or not), the economy and consequentially the markets will reflect that turbulence (or lack thereof).

You propose an interesting "take" on understanding the markets by understanding the "big" players (and their computers) actions underlying that market.

It is reassuring that microeconomic theory would seem to be alive and well.

The only people who can really exploit the patterns are the traders who move the markets.  For most others, the mental preparation, constant attention to the markets, loss due to bid-ask spread, and other factors turns what looks good on paper into a losing proposition.  One moderate loss, even short term can eliminate all your small wins. The chart patterns always appear better than they actually turn out to be. Think fundamentals, economic conditions, good quality index funds, and long term.

In terms of short term strategies, what makes this strategy any different from the 100's of other hindsight strategies that are developed day in and day out?

Ralph:

Saying you can't exploit the actions of "the traders who move the market" is like saying, if you live in a jungle, you can't exploit the actions of the elephants.

You can let the elephants clear a path through the forest for you.  Simply follow their lead (and don't get trampled in the process).

You definitely need to be quick on your feet and alert but, then again, that is a general requirement in life (not just stock trading).

If you are trading using "chart patterns", you are already a loser.  I learned over two/2 decades ago that two/2 dimensional charting is 60s or, at the latest, 70s technology.  I do assume when you are saying charting you aren't interpreting charts (visually) in n-dimensional space?

It is also the case that - if you are looking for some perfect mathematical "trick" to make money in the market (with), then you are missing the fundamental boat.  The overall market and the forces driving the whole market are always bigger than any individual (or even group of individuals) IN THE LONG RUN.

Scumbags, like Soros, may be able to create short term (temporary) dislocations in a market.  However, eventually, the markets will recover and resume their "fundamental" behavior (good or bad).

James - you said it yourself:

"The overall market and the forces driving the whole market are always bigger than any individual (or even group of individuals) IN THE LONG RUN"

"Long runs" can be applied to active trading as well. All strategies look good on paper, even when backtested and forward tested. The reality is totally different. Eventually if you do it long enough, the market will grind you out, i.e the more trades you do, the less your return will be (study).  Also, you are correct in saying we are not using 70's technology. There are too many things stacked against the average trader nowadays;  i.e institutional algorithms developed by the brightest and best, trading bots, and split second execution time which is NEVER available to most traders. That in itself is a big advantage since the institutional algorithms can look back and tailor their strategies to what other traders are doing.

At least in Blackjack, the rules are static.  With the markets the rules always seem to change.  The incremental gains possible vs. buy and hold, or portfolio management strategies would just seem to require incredible amounts of capital to make it worth it to beat the market, and at least for me, not worth it at all.

Many other options available, mostly thru ETF's which allow for various combinations of risk/return and for changing outlooks.

Ralph:

Not trying to bust your bubble but you are wrong on a number of points (commonly held myths).

First, if reality is "totally different" from your strategies, then you have the wrong strategies.  Strategies should REFLECT reality.  Dah.

Second, "there are too many things stacked against the average trader" is probably true because the "average" trader is an unknowing, inexperienced, amateur who ought to play craps in Vegas (and not the lottery, which is usually what they do instead).  You don't (or shouldn't) judge whether success is possible at something by "appealing" to what the "average" anybody can or can't do.  Of course, I am assuming you are not just ABOVE AVERAGE.  You are exceptional.  Right.

Third, frankly, it was my experience as an independent consultant that, generally, institutions (and those working for them) were by definition mediocre - even their best and brightest.  Most truly talented people can't tolerate the frustration of working for an institution.  I know I was constantly frustrated by the bureaucratic bullshit that passes for corporations (as institutions).  Of course, however bad (retarded) the private sector is, government in all it's forms is infinitely worse - even, their best and brightest who are often either ignored, passed over (for lack of seniority), or actually penalized (you can't fire government workers, generally).  That damn civil service is like tenure in the academic community.

Fourth, I agree reality definitely isn't static.  It is in that dynamic atmosphere that the individual "small" trader has a chance.  Larger organizations can't react as fast as smaller ones and smaller ones can't react as fast as the individual.  You apparently don't like change because you don't understand the PROCESS which underlies the SYSTEMS which are changing.  I generally do (at least enough to make money routinely and safely in the stock market).

Fifth, portfolio management is one of those bullshit concepts that have been sold by the large institutions to the "masses".

By definition, why would anyone want to "balance" one's "portfolio" with a bunch of sub-optimal stocks IF YOU COULD TELL A "GOOD" STOCK FROM A "NOT SO GOOD" ONE.  Again, I can tell the difference.  I could give you a bunch of examples (trades I have made).  One will suffice.

When Obama's Fed forced Suntrust to take "bailout" money when they didn't need it, Suntrust went thru a period where they were limited as to what they could do AND had to pay a bunch of fees they otherwise would not have had to pay.  Their tangible book value was around \$30 per share and they were selling as low as \$8 per share.  ANY MORON ought to be able to figure out that, unless Suntrust was going out of business - and they weren't, eventually their traded price would recover to at least the \$30 per share.  The key question was WHEN.  WHEN I found out that Suntrust was finally going to get out of the TARP program (free at last, free at last, thank God they were free at last) AND had a good idea when (roughly), I started buying their stock AS IT STARTED CLIMBING OUT OF THE BASEMENT.  i would buy it.  It would go up (the expected amount).  I would sell it.  It went down.  Then I would buy it again and repeat the process.

On the other hand, after trading it for a couple of times, my wife saw me making money on (and talking about Suntrust).  She asked me if I thought it was not just a trading opportunity but also a longer term buy.  I told her I thought the stock would go to AT LEAST \$25 per share.  She bought Suntrust around \$12 per share and held it (unfortunately too long - after it got to \$30).  Her hold wasn't a terrible mistake but the growth rate of Suntrust since hitting \$30 per share is not nearly as good as when it went from \$8 to \$30 per share.

I could repeat this story 100 times with other companies.

Finally, the "small" investor has an inherent advantage over the one who is managing a large pile of money.  It's based on the concept of liquidity.  You can't make a profit unless and until you can sell that stock that you bought for more money than you paid for it.

When large institutions get into and out of the market, the market "feels" the money coming in and going out.  When I (with my little pile of cash by their standards) do the same, nobody even knows I did anything.  Also, if one put \$10 million into some small companies (that would make a better trade than most of the bigger companies), the \$10 million would buy up all of the company's available stock.  It called capitalization.  As a "piss ant" investor, I have a whole lot more companies that I can put ALL OF MY MONEY INTO (to get a really cheap price) and, when I want to get rid of the stock, I can dump my whole position (in this piss ant company) and there will be plenty of buyers that I can sell my whole \$10,000 or \$100,000 position to.  LIQUIDITY RULES.

You clearly should stay out of the "trading" market.  YOU will surely lose all your money because you clearly don't understand what's going on (and how to make money routinely and reliably).  I been doing it (on and off) since the 70s on a routine basis.  So much for your theories.  Then again, I'm sure you don't have a degree in accounting, economics, political science, applied mathematics, business/IT/accounting work experience, and decades of first paper trading then money trading as I got older (and richer).

Well typed!  Yes, there is alot of investment marketing bullshit about "you can't invest! but give it to use and we can invest it."  Just listen to American Greed to hear the BS and how sociopaths use these tricks to convince you to give up your money to them!

Charles Biederman's markets studies center completely around liquidity and money supply and demand.

I have been doing it from the 80's as does James, however, we have all digressed from the original point of the post, "the market is very difficult to model and predict from a mathematical point of view."  The market has a dynamic which can be summarized as "it doesn't matter until it matters!" whic makes it very difficult to model the different focus points. . .

peace out!

James - All of the examples that you find always involve publicity concerning the 1% high and low tails where people make an incredible amount of money, or lose an incredible amount (usually at the hands of institutions).  I am more interested in the 99% of what people do, which also includes people with the advanced degrees and experience that you mention.

So much for anecdotal stories. There are numerous repeated studies which demonstrate my points.

http://faculty.haas.berkeley.edu/odean/

I can sent you many more papers which support this, if you would like.

Feel free to contribute any empirical evidence you have which supports YOUR points. I'm willing to listen.  Mostly I've seen 'advice' regarding personality traits, and discipline methodologies. Nothing hard-core.

I think actual empirical evidence is pretty hard to come by which supports active trading.

Ralph:

Since I don't know anything about your "background", I don't know where to start - except to say, the Suntrust example listed is TYPICAL of a trade(s) that I will do to make money.

OF COURSE, simply looking at tangible book value (or even a complete set of fundamental company based parameters is only a small (but important) part of my analysis BEFORE I start trading a company.

I would agree wholeheartedly with Terry's aphorism - trading (not only) can be hazardous to your (financial) health, but, for the average "smuck" who sees trading as a casual exercise that can be successfully done by untrained morons it is almost certainly a fatal (financial) disease.

Unfortunately, a lot of the "CRAP" pitched by the large financial houses about how to manage your money is also "pitched" in academic institutions.  They then teach that stuff as if it were gospel.

That is not to say that portfolio management (and related issues) don't deserve some comment any more than the concept of "financial planning".

However, as to financial planning, the best financial plan is to have enough money so that your earnings off that money exceed your current (and projected future) cash flow requirements.  Anything less and you are screwed (out of money).

That's why I always laugh when these financial guys try to convince some poor "smuck" who never saved enough money that they can bring peace and happiness to them in their declining years.  Most likely these "smucks" will end up dying for lack of medical care and be reduced to eating donated dog food (or on the public dole - assuming the US government and economy haven't yet collapsed and breached their unfunded promises to the citizens).  The financial failure (or breach of the promises made by politicians to get votes) of the welfare system in the US is a lot more likely THAN NOT.

A complete explanation of my stock trading systems (process) would actually require a book sized document just to describe it in technical terms.  Many of the technical terms themselves would also have to be "explained" to the average non-technical reader.  Also, I have discovered that many of my qualitative principles (in what I call my Pyramid of Principles because of it's hierarchical structure) are actually more important (from a process point of view) than any specific technical test or measure.

Philosophically, that shouldn't come as a surprise.  You can't quantify what you can't describe.  Another way of saying it is - you can have two/2 of something unless and until you have conceptually created (and often named) one of something.

If just plain old counting is subordinate to the process of human perception and language/expression (qualitative factors), then obviously that same argument would apply to the whole field of analysis (logic and epistemology) and mathematics as being  subordinate to "qualitative" factors (of which counting is only one).

Keep rowing.