madspeculator

Kiwi: Yes, I have done some comparative studies to determine the "usability" of my measures. I use the term "usability" instead of "predictability" for the following reason: In trading, people use the word "predictability" to mean prices going in the direction they expect the price to go. However, I define "predictability" to mean either (a) price going in the expected direction; or (b) price meandering around the area where the price first started. The reason I define "predictability" as above is because this event, which I call "positives", provides very little risk to a trade. However, the "false positive" (price going in the opposite direction than what was predicted), in my mind, is the true measure of risk for a trading model; my risk management is a function of my false positive analysis. Having said the above, my false positive percentages for raw measurement on delta relationships, value-area relationships, value-area and shape of profile relationships, and vwap-hvp (PoC) relationships hover around 50%, with vwap-hvp being around 45% (a lower false positive percentage is better). My measurements of imbalance gives me a false positive percentage in the low 30s. I used about 2 years of tick-data for this analysis. Note: I did not overlay other statistical transforms to the raw measurements to determine false positive percentage. Although that is what practitioners do in the industry to get an "edge", I was not mining for a trading model, and hence was not in my interest. Hope this answers your question.

I haven't figured out how to edit the post. So, I am posting a correction, although small, very important. Under the "Tools of Measurement" Section in author's work, I said: "The sum of these two parameters is the “real” order flow. All other volume is due to the activities of the algorithms (which the author considers to be noise)." This should read: The sum of these two parameters is the “real” order flow imbalance. All other volume is due to the activities of the algorithms (which the author considers to be noise) or balanced trade.

So far, we have made a case for a good theory to provide a framework for trading model research (Part I of this series), and provided a survey and critique of the existing theory of supply and demand (Part II of this series), and the theory of value (Part III of this series). In this final post, we discuss the theory of Market Action, and tools that can be created to measure it. Theory of Markets Action The “Kyle” model proposed in the field of market microstructure motivates this theory of Market Action. However, the reader should be aware that these two models are vastly different. The theory of Market Action is based on the following tenants (the conditions and consequence of each tenant are listed below it): 1. There are two categories of traders: liquidity providers (market makers), and liquidity takers (market takers). a. Existence of two categories does not prevent individual market participants from switching between being a liquidity provider and liquidity taker based on their trading needs; b. A consequence of this tenant is that competition exists only within members of each category and never between categories. 2. Both the category of traders are profit motivated. a. There is no quest for value in the market. 3. Liquidity providers react to order-flow from liquidity takers. a. Liquidity takers, on the other hand, trade for variety of reasons (“Mill Process”) b. This, along with the next tenant, is the reason for price movements. c. This implies that liquidity providers might capitalize at times of weak order-flow from liquidity takers to make a profit (e.g. triggering stops) d. However, when there is a strong order flow, going against such order flow will be to the detriment of the liquidity provider. 4. To liquidity providers, inventory management is of paramount importance. a. Inventory poses risk to liquidity providers; so, it is in the interest of liquidity providers to balance their inventory as soon as possible. b. The areas of accumulation and distribution are results of liquidity providers managing their inventory in response to anticipated or actual order-flow (from liquidity takers). c. The formation of high volume price (PoC or HVP) is also the result of this inventory management process. d. Liquidity providers usually increase their chance of balancing their inventory when price reaches certain points that are of interest to liquidity takers. i. This is the reason for price hitting certain well-known targets like previous day open, high, low, close; high volume price, PoC, etc. ii. However, competition within liquidity providers tends to destroy tradable patterns or prevent price from hitting those well-known targets. There are four parameters to this theory; they are: demand and supply of liquidity provider, and demand and supply of liquidity takers. Since this theory depends heavily on liquidity provider inventory, we need to explain what this “inventory” really is. This brief digression will help us explain and define the term “inventory”: Let us assume that a trader needs to transact a large transaction. There are three ways this trader can accomplish her goal: (i) By means of a market order. Since the trader’s transaction will move the market, the average price received by the trader for her transaction might not be optimal. So, this might not be the preferred method. (ii) By means of a limit order. This is a little better than a market order, but the trader has to be content with not getting her entire order filled (if the market moves away, because other traders notice the “support” or “resistance”, then the order will not be completely filled). This is not an acceptable situation. (iii) By means of some algorithm to execute this order (a.k.a algo trading or program trading). The providers of these algorithms, usually the big banks or hedge funds, guarantee complete “fill” of the traders’ orders at a “benchmark” price (the details of this are not essential for our present purpose). This methodology of trade execution is usually preferable to the large trader. These algorithms, in the process of filling a trader’s order, end up buying or selling more quantity that is really needed. This excess quantity, in the hands of the algo-trading providers, then becomes “inventory”. It is this excess inventory that the liquidity providers (in this case, the algo-trading providers, not the trade whose order is being executed) need to manage. [Note: We are not concerned about client orders that might in a bank’s book. We are only concerned about the excess inventory these banks acquire in the process of filling such client orders] An astute reader will immediately recognize that the Wyckoff theory of markets is nothing but a sub-set of this theory – Wyckoffian parameters are either higher order (cause, effect, effort, result) or aggregate (demand and supply) of this theory’s parameters. This theory strives to explain a lot more of the observable phenomena in the markets, which remain unanswered by Wyckoff’s theory. A note on the applicability of this theory: The author is confident that this theory can be applicable in all markets where T&S and volume information is dispersed. However, the author has tested this theory only in limit-order markets (an example of a limit-order market would be the ES on Globex; most of the non-option electronic markets are either limit-order markets or hybrid). Tools of measurement All existing tools discussed under Wyckoff section can be used to measure the theory’s parameters either in aggregate or indirectly (using second- and higher-order parameters). Volume- and market-profile tools can be used to identify (measure) liquidity providers’ inventory adjustment zones (a second order parameter, which presents itself as a bulge in the profile). [Please note the first standard deviation from PoC has not practical value.] It must be noted that a profile could contain multiple inventory adjustment zones (double distribution days) or could include multiple inventory adjustment zones that occur around the same price range but at different times (i.e., the profile could show just one “value area” but in reality could be multiple inventory adjustment zones spread out temporally but in the same price range). However, the advantage of this theory is that new tools can be developed to further one’s understanding of the market, and more specifically, the operations of the liquidity providers – a very valuable information to a trader. Three examples of such measuring tools are presented here: 1. Reference is made to this thread by UrmeBlume (UB). The author is not aware of any particular details of UB’s work, but UB’s explanation of his work makes the author believe that he could be measuring the activities of liquidity providers and liquidity takers. 2. FulcurmTrader’s (FT) thread gives the author the impression that FT is tracking the activities of liquidity providers. 3. The author’s own work presented below tries to estimate the demand and supply of both the liquidity providers and liquidity takers. By monitoring the demand and supply of liquidity providers, the author is able to notice the inventory adjustment process without the aid of the profile (which has a tendency to include a lot of noise, therefore less accurate – a major problem which is a result of the profile’s construction technique). Measurements of parameters for different time frames and periods are captured in the attached charts (first two are 30-min charts, and the last two are day charts). NIP measures the inventory held by liquidity providers, and NOP measures imbalance in the liquidity takers’ order flow. The sum of these two parameters is the “real” order flow. All other volume is due to the activities of the algorithms (which the author considers to be noise). [As the motivation for showing these charts is to provide examples of possible new methods of measuring the theory’s parameters, the author will refrain from analyzing these charts, although such an urge is very much present.] A quick note on these charts: These charts were not handpicked. The author used whatever data was available on his laptop to create these charts for this article. Concluding comments As can be seen, the extent to which one can research and extract parameters depends on the quality of the theory of markets one subscribes to. This is by no means the one and only applicable theory to market action. What is important is having a theory as a base for one’s research. Should you not have a theory that challenges you to go that extra-mile to generate your edge, please feel free to use the one presented here, of course, only if you are comfortable with its tenants. The author does believe that the tenants of this theory are strong enough that it will survive the test of time, and that new tools will be invented in the future that will provide more accurate measurements of the parameters presented in this theory, thereby making the present day tools outdated. If one decides to undertake research based on the above-mentioned theory, please note that one cannot split ‘bid’ and ‘ask’ as is proposed/done in the industry today in order to measure all of this theory’s parameters. One has to perform the analysis from the perspective of a liquidity provider (Ask the question: If I were to end up with inventory as a result of me trading a client’s order, how will I trade? This, I hope, will lead you to an acceptable answer, as it did to me). Once one figures this out, the land is theirs to conquer! It is suffice to say that there is no holy grail in trading, and this author doesn’t consider the author’s tool to be the holy grail either. As was said earlier in this article, the lack of accuracy in measuring a theory’s parameters will always make trading a game of probabilities; a good tool, however, can give one the much-needed “edge”. Hope your research helps you create a new tool of measurement, which might result in you harvesting multiple trading systems. A short note on why I wrote this article My training in both hard and soft sciences, and my exposure both to the “buy-side” and “sell-side” of the business made me evaluate, question and/or reject the theories that underlie the publicly available measuring tools. When it was time for me to develop a “quantitative” model that could be used to generate trading strategy(s), I had to create a theory that was capable of explaining all observable phenomena in the market. The knowledge I gained during this process was invaluable; I wanted to share that knowledge and provide a helping hand to those interested in “quantitative research” by giving them a stepping-stone for their work – a stepping-stone that was not available to me when I started my work. I am not a vendor; I don’t plan to be one; and, no part of my work is for sale. Although I might be able to answer questions related to this proposed theory or the research process, I am, due to various constraints, unable to answer inquires related to (or disclose) the construction of my measuring tool. For those interested in the tools I use to perform my research: I use the statistical package R. I do my research using R for two reasons: (1) it has good data manipulation and visualization capabilities. The graphs were created using ggplot2 package in R. R also has the ability to connect to Gobi to visualize multiple-variable/multivariate data; and (2) it helps me perform statistical analysis on new trading strategies with ease. Thank your for coming along for the ride; hope you got something useful from it.

Thank your, BlowFish. My understand of Wyckoff is that 'cause' results in an 'effect'. The 'cause' is accumulation and distribution areas, while the 'effect' is the price movement due to the 'cause'. Wyckoff did use this as parameters to his system, since this is what his theory measured using the PnF charts. Your are correct in that 'effort' and 'result' are measured by the price + volume chart in the Wyckoff theory. That is precisely why I consider 'effort' and 'result' to be parameters in the Wycokoffian theory. I might have a different taken on this. You will see my line of thinking when I present the proposed theory of market action in Part IV of the series.

Having discussed the shortcomings of the existing theory based on supply and demand, we move on to survey and critique the other theory: the theory based on value. Theory based on value In the early to mid 1980s, J. Peter Steidlmayer developed a revolutionary technique to visualize price action. He (and CBOT) named it Market Profile. Based on the observation that the profile had a bulge, he postulated a theory based on value (the term “value investing” – in the likes of Ben Graham and David Dodd – was in vogue during that time), and thus the auction market theory was born. The major tenants of this theory are: 1. There are different time-frame traders in the market. 2. Traders seek value; Price is all but an advertising mechanism in search of value. 3. Longer time-frame traders don’t interact directly with each other but through short time-frame traders. 4. Market moves only as a result of convection by a certain group of long time-frame traders. This theory was born from empirical analysis based on the measuring tool – the profile graphic. This theory has five parameters: value, demand and supply of longer time-frame traders, and demand and supply of short time-frame traders. Tools of measurement Since the theory of value was born from the empirical analysis of the results from market profile, market profile became the primary tool of measurement. [Although the validity of construction of market profile might be a point for debate – should all the data points on a timeframe be given equal weight – such discussion is out of the scope of this post.]. As more information started decimating from the exchanges, volume profiles are being used to measure value. Unfortunately, the market profile and the volume profile measures only one parameter of the theory – value. There exists no publicly available tool, as far as the author is aware of, that measures the other parameters of the theory, namely, demand and supply due to long time-frame and short time-frame traders. However, traders use tools developed to measure Wyckoffian parameters to predict the presence of the type (buyers vs. sellers) of long time-frame traders based on the location of such activity in relation to the value area of the profile and the resulting movement in price (initiating vs. responsive actions). Side note: Although Steidlmayer advocated that value be measured only in “balanced” distributions, Dalton and others advocated the measurement of value even for skewed distributions. Steidlmayer emphasized that people should “find” the “hidden” value for skewed distributions by assuming that such skewed distributions eventually becomes a balanced distribution (interested readers are referred to his work on minus distribution, and more recent work on supply-based markets, and value-based markets). Critique of the theory based on value Of the two theories discussed in this article, the theory based on value rests on a very week footing. This theory hinges on the concept of value, yet the very definition of value is questionable. Value, according to this theory, is defined as the first standard deviation from the Point of Control (PoC). Why the first standard deviation? No answer is provided. If the definition of value according to this theory is indeed valid, the question arises as to why prices deviate from value. The answer provided is that perception of value amongst market participants change, thus changing value. This further raises another question: why do perceptions of market participants change so often – sometimes within a day or even when no substantial news comes out? No answer is provided to this question. Nor is there any plausible answer available in the larger financial or economic literature. Further, an astute observer will observe that the so-called value changes based on the duration in which the profile is constructed. This points to the possibility that value is duration (time) dependent. The consequence of this observation, according to this theory, is that shorter time-frame traders have a different notion of value, compared to larger-time frame traders. Think about the consequence of the above statement for a moment. If an individual participant decides to change from being a short time-frame trader to being a long time-frame trader sometime during the day, her notion of value changes at that moment of decision. In other words, an individual’s perception of value changes because they change their trading time-frame. This definition of value clearly deviates from the standard economic definition of value (or equilibrium price) or individual’s utility value. The next criticism is on measuring the so-called value using PoC as the starting point. Careful analysis of PoC suggests that PoC in a volume profile is formed around the area where a few large players transact large amount of trades. This would indicate that large players are responsible for defining value. Argument along the lines used above can be used to discount the use of PoC to calculate value. In the case of market profile, time determines PoC. Although Steidlmayer claims to have “removed time from the equation”, it is time that is being show in the horizontal axis of the market profile. Such a dependence of value on time, irrespective of the amount of transactions transacted at value, once again deviates from the standard economic definition of value. So, is “value” in this theory really value in an economic sense? Or is it a misnomer for some other phenomenon? Why is this question important? The only way for one to further or fully understand any concept is when words are not overloaded (using a well understood word to mean a different thing). This means that all phenomena should be properly identified and named. For example, if one calls a house a dog (for which a definition already exists), then the listener’s understanding of that conversation will be markedly different than if a house was called a house. If words are indeed overloaded, proper definition for the word should be provided. Further, an overloaded word prevents us from determining if that word is really a principle component of the system or is a second order component of the system – is value a principle component or a second order component? As indicated earlier, in our field, where accuracy of measurement is a problem, measurement of second order components produces larger errors. The author contends that “value” according to this theory is indeed a misnomer (for a different phenomenon) and is a symptom (second-order component) of the trading operations of market makers (principle components being demand and supply of market makers). The proposed theory of market action identifies this misnamed phenomenon. The lack of publicly available measuring tools to measure long-term and short-term traders’ demand and supply prevents us from validating tenants one, three, and four. Further, it appears to the author that tenant number three might be more applicable in “pit-trading” markets than in the present day electronic markets. A theory or hypothesis should not be rejected solely based on the inability to measure all its parameters – in this case, long time-frame and short time-frame demand and supply schedules. People will invent new techniques to measure those variables at some point in time. However, a theory or hypothesis should be rejected if it is incongruent or its predictions contradict other well-established theories. Such is the case for the theory based on value. We will reject it. It has to be noted that existence of tradable patterns does not support a theory. So, traders should heed caution that such tradable patterns might disappear. However, the local highest volume price (called PoC) has short-term significance as will be discussed in the proposed theory of Markets Action. As a side note for those interested: Steidlmayer’s new work includes volume analysis. Further, he is now championing a new theory based on price-time measurements. I don’t think he considers PoC to be “value” anymore. In fact, he doesn’t use the words PoC or “value area” anymore, but uses the term “zero-line”. This of course is the mark of a true great trader – to understand one’s system and why it works (theory of markets), and when the theory behind one’s system is faulty, the system doesn’t work as expected (although the system might still provide tradable patterns); move on and search for a more applicable theory of markets to conduct your research, while still, may be, trading those tradable patterns until they fade away! In the next – last and final – post, we will discuss the proposed theory of Market Action and present examples of new measuring tools.

Richard, I should have made the context clear. The target of that statement was people who exclusively trade one instrument. As I had stated in Part I of this article, inter-instrument activities (like pair trades, arbitrages, etc) are more about efficiency than about market supply and demand, and hence discussing those types of trading was outside the scope of this article. If a person (naked) shorts way out-of-money options to profit on time premium, they would be an exception to my statement. However, a person trading naked options to profit from price action of the underlier would benefit from this article. (Note: if one tries to profit from theta by maintaining a fully hedged position, I would consider that inter-instrument activity. I would also consider gamma or vega trading to be a inter-instrument activity) Thanks for raising this point.

In the previous post, we discussed the need for a good theory that provides a framework for one’s research to generate sustainable trading models. I also promised a quick survey of two existing theories and their shortcomings. In this post, we discuss the first theory: Theory based on supply and demand. Before we proceed, we take a short digression to clarify certain points. Both hypothesis and theory try to describe a phenomenon. Take the Dow “theory” for example. It states that prices moves in waves. This is an observation. Observations are neither theories nor hypotheses. Since Dow did not describe why prices move in waves, we don’t have any parameters to measure. What separates a theory from a hypothesis is the ability to measure. Take the Fractal Market Hypothesis (FMH) for example. FMH describes the reasons for market action. Unfortunately, at this present time, there is not tool to measure all the parameters in FMH. Until such tools are found, FMH will remain a hypothesis. A note also has to be made on the falsify-ability of a theory. In order to falsify a theory, the measurements have to be accurate. Unfortunately, in the field we are in, accurately measuring the [hidden] parameters of a theory is not always possible. This is the reason why a theory in trading can never be falsified based on measurements alone; this (accuracy of measurements) is the same reason why trading will always be a game of probabilities. In fact, if all hidden parameters of a market can be accurately measured, the market will cease to exist. Just because a theory cannot be falsified based on accuracy of measurements does not make it a hypothesis. If trading is a game of probabilities, the goal, then, is to get one’s probabilities higher than the rest of one’s peer group. This can be achieved by increasing the accuracy of measurements of a theory’s parameters – the reason why Wilder and the turtles were more successful than the Edwards & Magee type traders. But what if the accuracy of measurements amongst one’s peer group is almost same? Then, the only way to increase one’s probabilities is to improve the quality of the theory: a theory that uncovers hidden, measurable parameters that better describe the actions of the market – a topic we are concerned about. Lets get back to the survey. Theory based on supply and demand: Wyckoff’s theory of market action and its variants More than 150 years ago, in 1848, John Stuart Mill wrote a vivid description of speculative market behavior: ”The inclination of the mercantile public to increase their demand for commodities by use of all or much of their credit as a purchasing power, depends on their expectation of profits. When there is a general impression that the price of some commodity is likely to rise, from an extra demand, a short crop, obstruction to importation, or any other cause, there is a disposition among dealers to increase their stocks, in order to profit from the expected rise. This disposition tends by itself to produce the effect which it looks forward to, a rise of price; and if the rise is considerable and progressive, other speculators are attracted, who, so long as the price has not begun to fall, are willing to believe that it will continue rising. These, by further purchases, produce a further advance: and thus a rise of price for which there were some rational grounds, is often heightened by merely speculative purchases, until it greatly exceeds what the original grounds will justify. After a while this begins to be perceived; the price ceases to rise, and the holders, thinking it time to realize their gains, are anxious to sell. Then the price begins to decline; the holders rush into the market to avoid a still greater loss, and, few being willing to buy in a falling market, the price falls much more suddenly than it rose.” It is this description that Richard D. Wyckoff tried to formalize in his theory (those interested in his theory are referred to the Wyckoff forum). His theory has six measurable quantities: demand, supply, cause, effect, effort, and result. [Please note that Wyckoff also provided “how-to” instructions; our immediate purpose is not to survey them.] A note on recent developments in behavioral finance: Finally, the academic community has decided to pay attention to the above-mentioned “Mill Process”. The result of their attention is the field of behavioral finance. As it stands now, the tenants of behavioral finance can be viewed, among others, as explaining the demand and supply components of the Wyckoff theory. So, for the purposes of this post, we can “roll” those models into the Wyckoff theory. The same argument can be made for the theory of reflexivity, a concept borrowed from the field of social sciences and applied to economic by George Soros. Tools to measure Wyckoff’s theory of market action The word “measure” is used very loosely in this post. This word should be read not necessarily with the rigor one would expect in a quantitative measure but as a synonym of estimate in a heuristic sense, wherever applicable. Wyckoff himself advocated the use of then existing tools to measure [the variables in] his theory: bar charts and volume to measure supply, demand, effort, and result; and PnF charts to measure cause and effect (a.k.a accumulation/distribution areas) Tom Williams later introduced VSA, a different interpretation of the bar charts and volume, to measure the demand, supply, effort, and result components of the Wyckoff theory. He too advocated the PnF charts to measure cause and effect. The works of Robert D Edwards, and John Magee used bar charts in-by-themselves to measure supply, demand, cause, and effect. The assumption they worked on is that price action in-by-itself captures the variations in supply and demand, and that using volume in addition to price action was unnecessary. According to them, price formation (or patterns) provided clues (measures) of market conditions (demand, supply, cause, and effect of the Wyckoff equation). Our intent is not to argue for or against the merits of their argument, but to provide a survey. The work of J. Wells Wilder, published through his book in 1978, revolutionized the field of measuring tools by introducing the concept of “indicators”. “Indicators” are nothing but statistical transforms of underlying data streams. Over the course of years, different statistical transforms have been developed that helped provide quantitative measures of all the variables in the Wyckoff market theory. A note on the usage of the word indicator in this post: By definition, the purpose of an indicator is to indicate. However, an argument can be made that consecutive price bars (and or volume bars) in-by-themselves are indicators as they indicate the direction of price action (a single price bar doesn’t indicate anything useful, but it is by comparing successive price bars that an indication is provided). Further more, when we get to the section on Market Profiles, the word indicator become over-loaded: a single market profile graphic is both an indicator – it indicates where the value area is, which in-by-itself is useful – and a summary statistic of the underlying data just like a single price bar. So, to avoid confusion, the word indicator will not be used in this post, instead the word statistical transforms will be used. As can be seen in this section, a plethora of tools, both quantitative and otherwise, are available to measure the variables in Wyckoff’s theory. It has to be noted that traders who don’t trade using the ideas of market profile, knowingly or unknowingly adhere to the theory of supply and demand as presented here. Short-coming of Wyckoff’s theory Wyckoff’s theory is a good starting point. However, it has one glaring shortcoming. In order to fully understand a system, the system has to be expressed in its principle components. Only the principle components of the system are allowed to be influenced by external stimulus. In the case of Wyckoff’s theory, when we ask the question “what is the reason for demand and supply?”, we get an answer “because people buy and sell”. People buying and selling (external stimulus) causes demand and supply (principle components) in the system. However, when we ask the question “what is the reason for effort and result, or cause and effect”, we get an answer “because of the actions of composite operator”. Clearly the parameters effort, result, cause, and effect are symptoms (second order parameters) of the composite operator’s actions (which causes composite operator’s demand and composite operator’s supply within the system). Why is this important? In a system like ours, where accuracy of measurement is a problem, second order parameters over amplify measuring errors resulting in, not increasing one’s trading probabilities, but plausibly reducing them. A new theory should take into account the trading mechanisms of market makers (or commercials or liquidity providers or whatever name one prefers). Adding this variable to the theory of market action creates a principle component, if properly measured, can provide clues to the actions of the market makers – a very valuable information to traders. We, then, now longer need to measure the second-order Wyckoffian parameters cause, effect, effort, and result. A concluding note, in support of Wyckoff: Wyckoff designed his theory to promote his how-to methodology to retail traders when the information available to retail traders was very limited. The author contends that Wyckoff had not other option but to include the parameters cause, effect, effort, and result in order to educate the retail traders in his how-to methodology. The author applauds his efforts. Given the information we have today, we can do better. In the next post, we will discuss the theory based on value (Market Profile) and its shortcomings.

My next post should provide the answer to your question.

Introduction Every trader reaches a point when he strives to improve upon his trading – the unsuccessful in order to make money, the successful in order to improve their bottom-line, and the “were-successful” in order to start making money again. It is at this time a fundamental question is raised: “Why is the market doing what it is doing?” Although this question comes in different forms (“I wish I knew what was going on in the market at this point”), the underlying theme is the same: The quest to understand market internals. A section of the trading population tries to avert this question by taking the so-called “self-analysis” route. Although self-analysis can be useful for the growth of a trader, such an approach in-by-itself, without a good understanding of the market internals, is almost useless; invariably, they will end-up in the other camp. Within the other section of the trading population, some quit, while some resort to research to develop a better trading model. This article is targeted towards those who resort to research to improve their trading. Peter Drucker, the famed management consultant, once wrote that there were two kinds of books from which knowledge can be gleaned: the what-to book, and the how-to book. Within the context of trading, the how-to book would teach one how to trade. This site has a number of good posts on that subject (e.g., JPerl’s posts on trading with market statistics); this post is not one of them. This post is about the what-to – what should one do in order to understand the market internals? Need for a theory of market The quest to understand market internals cannot start without a theory. The problem with developing a trading system without a supporting theory is that such systems will stop working when the “market conditions change”. A good theory of market action should be able to prediction such “market condition changes” (prediction not in terms of exactness – which no one can do – but as a way of market operation). Statistical model builders who do not have a market theory to back their models up are (or at the least, should be) keenly aware of this problem. [Arbitrage models also stop working at some point in time, but for a different reason: competition leads to efficiency. Such a theory is very well established in the finance literature, and is out of scope of this discussion] A discretionary trader trying to develop a system cannot do so with out a theory; nor can a systematic trader. Once a “tradable pattern” is found, the type of validation performed to determine long term viability of such “tradable pattern” might be different between these trading groups, but the type of research performed is the same – one rooted in one’s understanding of the workings of the market. The expectation that computers can automatically mine meaningful patterns from complex financial time-series without human guidance is all but based on naivety. [The author is not interested in a debate on which type of trading is better: discretionary or systematic. People do what they do because they are comfortable with what they do.] In no other field is a good theory more important that in the realm of trading. Unfortunately, this is the very field where a good theoretical underpinning required to conduct structured research is lacking. Without such theory, the sheer amount of data will overwhelm the researcher. Survey of existing theories According to Wikipedia: A theory, in the scientific sense of the word, is an analytic structure designed to explain a set of empirical observations. A scientific theory does two things: 1. it identifies this set of distinct observations as a class of phenomena, and 2. makes assertions about the underlying reality that brings about or affects this class. In other words, a theory not only describes an observable phenomenon but also provides means to measure it. Without proper measurement, a theory cannot be falsified – a prerequisite proposed by the great philosopher Popper. Theories that cannot be measured are hypotheses; hypotheses are not very valuable for traders. It is not that the field of trading is devoid of theories; it has its own share of theories. However, most of them are really hypotheses. Of the few theories floating around, they either do not provide enough detail to further one’s understanding of the market or that they are simply invalid. So, our first order of business before delving into our own theory of market action is to survey the existing theories of market internals. In the light of the above definition, some so-called theories (like the Dow theory – which doesn’t describe why price moves in waves, or fractal market hypothesis (note not a theory) which doesn’t provide means of measurement) will not be discussed in this section. There are two theories that are worth mentioning: 1. Theory based on supply and demand (e.g. Wyckoff’s theory of market action); and 2. Theory based on value (e.g. auction market theory, market profile) In the next post, which I hope to post sometime towards the early part of next week, we will conduct a quick survey of these two existing theories and their shortcomings.

Clmacdougall: Would you mind sharing with us excerpts of your conversation with UB when you find time? If sharing such information would infringe upon UB's intellectual property rights, I can understand you not wanting to share that information. Thanks.

I understand why you wrote that, but the real one goes like this: He who knows not, and knows not that he knows not, is a fool...shun him. He who knows not, and knows that he knows not, is willing...teach him. He who knows, and knows not that he knows, is asleep...awaken him. He who knows, and knows that he knows, is wise...follow him. For other variations see Knows and Knows Not -Madspeculator

I agree with you that calling the new format "more accurate data" is not correct. However, I disagree with you that the new format " ..is less accurate ..." and that we have "lost transparency ..... creates ... uneven playing field for retail investor." CME's new format just represents the "other" side of the coin. As you might be aware, Globex is a limit-order book market. So, how does one report trades in a limit-order book market? One could report market orders or report limit-orders that fill market orders. Depending on who you are, the type of reporting might mean different things: If you are a large trader, receiving data from the limit-order book's fill gives information about the liquidity (and type of liquidity) present in the market. This information on liquidity is very important to a large trader because she can be comfortable knowing that her size will not move markets. Moreover, the type of liquidity present at any point in time will dictate the strategy employed by her to execute her orders. If you are a trader who depends on reports of market order fills, this new format will not enable you to see the size of the market order. Because of this reason, you claim that you have lost transparency, and that this has created an uneven playing field. Well, this claim, although definitely NOT objective, depends on one's perspective. A large trader, operating in an exchange that reports fills on market orders, might think this type of report is not fair to her because exposing her size provides opportunities for other traders to front running her orders. So, the large trader has no other option but to split her market orders into small chunks. This is not "hiding true intentions or manipulation" as you claim (which sounds derogatory) but is something necessary for the large trader to do in order to avoid ending up with "bad" fills. In spite of doing all this work to prevent exposing her orders, the large trader has no clue about the liquidity in the market which is the most important thing to her - how unfair!!!! If you are an exchange then you want to report trades in the new format. An exchange can exist only if the markets it operates can provide liquidity to traders. The more the exchange does to increase liquidity in its markets, the better off the exchange is. So, this change in reporting format by CME is in line with what CME has to do to create more liquidity in its markets. Your claim "...I fully believe it will only get worse as open outcry disappears entirely" has no merit unless you are privy to information that most of us don't have. I don't know what your background in trading is and how much you really understand how institutions and market makers trade, but looks to me that you are making this an emotional issue for yourself by giving this change -- the way CME now disseminates data -- more weight that it really deserves. May be your trading is TOO dependent on detecting "large" trades on the tape, in which case I can see why you might be upset with this change. Time to change your trading strategy, may be? All the best. Regards, MadSpeculator

Hello Fellow Traders: I am a futures trader who trades at CME/CBOT (cmegroup) and monitors ICE. I currently use CQG as my datafeed. Unfortunately, given my cashflow situation, CQG is getting too expensive. I am a swing trader: I need real-time data and daily/weekly historical data (both price and volume). As I dont use any indicators in my trading, the charting package provided by the datafeed provider is not important for me. I was wondering if any of you would recommend a good datafeed (with API support) that is stable and reliable -- I was leaning towards esignal but I read that their daily volume info lags by a day, and they have roll-over and volume issues with continuous contracts? Or is there a list of datafeed providers "somewhere-out-there" that I can use to evaluate datafeed providers? I would be more than happy to evaluate the providers if there is an already available list. I look forward to your response. Thank you. Regards, Madspeculator. P.S: Soultrader: I am not sure if this is the correct place for this thread. Would it make sense to create a datafeed fourm under "Trader Resources" section where fellow traders can comment on their experiences with their datafeed(s)? Personally, I would like to see reviews of datafeeds separated out from the software provided by the datafeed providers (e.g., CQG's datafeed from their charting package; or esignal's datafeed from their charting pacakge).