NetTecture

No, because they CAN NOT EXIST. If you want to BUY with a limit HIGHER than the last price... you get.... EXECUTED. Nothing is left to be put into the ladder.

Slippage is lowest in the ES - obviously slippage is a functoin of order volume to market deph. The YM can be really bumpy around news - I have seen jumps of 50 points there as spikes, while the ES barely moved. Mostly because while the ES gets thinner during news times, the YM market sometimes gets razor thin. Which makes stop "explosions" more likely.... I have seen the YM bid having less than 10 contracts over 5 ticks totally - that is how thin the market got. On the ES, below 100 for ONE price is really thin.

::or am i wrong in that assumption Exactly that you are. Every relevant ORDER has to be filled. Not every step may have orders, and expecially not in enough volume to get you executed. I suggest watching the YM futures contract - it is not exactly a HEAVY market compared to the ES. Then watch before major news, and you will be surprised. I have seen the next 5 steps of the latter in each direction consist of less than 10 contracts TOTAL, normally you have 40-50 on one price each. If there is noone on the other side, for the matching price, then the price can gladly move without.... any trading.

You have a link to the discussion? I can not find it on the mypivots forum.

Ok, status update Change of hearts. Well, not really. * Data collector running since yesterday around 1900 UTC (for those now knowledgable: That is Greenwhich Mean Time WITHOUT summer/winter time - computers use that internally). * The log file so far has the form like 009-08-24 04:26:24.149830 ZDU9 CME Bid 0 9534 009-08-24 04:26:24.149830 ZDU9 CME Bid 10 9533 009-08-24 04:26:24.150741 DDU9 CME Ask 2 9572 009-08-24 04:26:24.150741 DDU9 CME Bid 0 9530 009-08-24 04:26:24.150741 DDU9 CME Bid 0 9547 009-08-24 04:26:24.150741 DDU9 CME Bid 2 9546 009-08-24 04:26:24.154835 ZDU9 CME Bid 2 9539 009-08-24 04:26:24.154835 ZDU9 CME Bid 12 9538 009-08-24 04:26:24.154983 DDU9 CME Bid 3 9529 009-08-24 04:26:24.155119 YMU9 CME Bid 4 9546 009-08-24 04:26:24.155385 ZDU9 CME Bid 0 9547 009-08-24 04:26:24.155385 ZDU9 CME Bid 1 9528 009-08-24 04:26:24.157930 DDU9 CME Bid 0 9548 009-08-24 04:26:24.157930 DDU9 CME Bid 2 9547 009-08-24 04:26:24.157930 DDU9 CME BestBid 2 9547 009-08-24 04:26:24.158331 YMZ9 CME Bid 2 9487 009-08-24 04:26:24.158331 YMZ9 CME Bid 3 9486 009-08-24 04:26:24.158331 YMZ9 CME Bid 0 9384 009-08-24 04:26:24.158331 YMZ9 CME Bid 1 9487 009-08-24 04:26:24.158331 YMZ9 CME BestBid 1 9487 009-08-24 04:26:24.158483 ZDU9 CME Bid 0 9548 009-08-24 04:26:24.158483 ZDU9 CME Bid 2 9547 009-08-24 04:26:24.164320 DDU9 CME Bid 0 9529 009-08-24 04:26:24.172657 ESU9 CME Bid 91 1030.75 009-08-24 04:26:24.172657 ESU9 CME Bid 100 1030.5 009-08-24 04:26:24.175702 ESU9 CME Bid 76 1029.75 009-08-24 04:26:24.175702 ESU9 CME Bid 128 1029.5 009-08-24 04:26:24.179895 YMZ9 CME Bid 0 9482 009-08-24 04:26:24.179895 YMZ9 CME Bid 10 9480 009-08-24 04:26:24.183040 ZDH0 CME Bid 0 9389 009-08-24 04:26:24.183040 ZDH0 CME Bid 5 9386 009-08-24 04:26:24.183040 ZDH0 CME BestBid 5 9386 009-08-24 04:26:24.199314 ZDZ9 CME Bid 0 9483 009-08-24 04:26:24.199314 ZDZ9 CME Bid 1 9481 009-08-24 04:26:24.199314 ZDZ9 CME BestBid 1 9481 009-08-24 04:26:24.200362 6JU9 CME Ask 21 0.01055 009-08-24 04:26:24.200362 6JU9 CME BestAsk 21 0.01055 009-08-24 04:26:24.200463 6JU9 CME Bid 18 0.010548 009-08-24 04:26:24.200463 6JU9 CME BestBid 18 0.010548 009-08-24 04:26:24.220230 6JU9 CME Ask 1 0.010549 009-08-24 04:26:24.220230 6JU9 CME Ask 0 0.010554 009-08-24 04:26:24.220230 6JU9 CME Ask 83 0.010552 009-08-24 04:26:24.220230 6JU9 CME BestAsk 1 0.010549 It is slightly bad: the hour is in 12 hour format instead 24. Not too bad for my purposes though. Main problem, though: it is already around 800mmb big I wont have the space for a whole week on the particular drive. I will stop collecting at the end of the day when the ES has closed or tomorrow - depending on disc usage today. I will then finish creating the other files needed tomorrow (mostly the descriptions and pricing steps) and make the archive available. I THEN go on with a binary representation, based most likely on delta storage (after all, things do not change that much, so I can but the price in one byte easily most of the time, between prints in one symbol). The idea is to store information "per symbol" in time slices of maybe 5 minutes (or more, depends on the amount of data I reallly get) in binary fields . And will see what loading that into my SQL Server says... (which has a lot more space than the drive set aside NOW for the log - I seriously did not expect THAT much data). The textual log file is simply waaaaayyyy tooo large. I will possibly reduce the granularity on the timestamp to about MS resolution. May be less. I do not really see a need for a better granularity than about 25ms (which is what NxCore uses, too). I will most likely start collecting again next week (I actually want to make full collection for my own purposes starting 1st of September), but I may filter the data - I simply do not need data on virtual instruments (suchj as spreads) And have not exactly a large need for options, so I may filter out the order book there, keeping only best bid and ask For those interested, btw: CPU utilization on my collecting station is really low so far, and network IO is around 1.6 megabyte per minute. That is around 200kbit I Post another update after market open - that is when it gets interesting.

No, not tried. Would be interested in it, too, though, if for anything just for the knowledge

Hello folks As part of my development process for a trading platform, I am testing my data collection system next week. This is a stress test and will take 1-2 weeks - I can not rule out I have to restart of get a crash first week, and I Want a complete week sample. Anyhow, I will collect a complete data feed from the CME, electronic trading (basically: the same you get via zen-fire, just COMPLETE, not filtered like zen-fire is, and not limited by symbol). This will include: * Preopening, openeing, closing etc. information * Trades, obviously * Best bid, Best ask, Bids AND Asks, including their invalidation (setting the bid / ask volume to 0 when the price moves out and the exchange does not track them. This will include the following instruments: * Naturally all futures. * All options, are published by the exchange. This is the majority of data. * Virtual instruments, such as spreads trades that are done on the exchange level. Most people won't know that but you can execute those on the exchange, which guarantees the spread value then. Those CAN be ignored - spread executions also show up as the separate leg trades in the instruments. The timestamps are in microsecond format from the original data capture equipment. The stream is totally unedited. It does NOT include cancels an corrections to my information. I record it to get an idea of the volume, analyse that my data capture approach and get an idea how to actually process that hugh amount of data - as a stress test for my equipment. Anyone interested in a copy (ONLY for the complete data) can contact me via PM. This is initially a free offer, but if the volume goes too high up I may have to ask for a small contribution for the data transfer amounts. Can not have 20 people downloading 50gb or so for free We talk of SMALL here - basically covery my traffic costs. Not sure how large the archive will be - ask me in 2 weeks. Anyone else is free to put that archive up on peer to peer - I even may do so, but with a very limited bandwidth The file format will be tab separated windows text file (CRLF delimiters), naturally compresed, most likely with 7zip Expect SIGNIFICANT Archives. We talk of 30.000+ lines PER SECOND in the the hot phases. Even during the night, often, the lines are not readable on a text output, because every trade results in a lot of bid adjustments. It will contain two file sets - one with the exchange prints, one with instrument metadata (name, allowed prices etc.). I may actually have to split the allowed prices out into a third file for technical reasons. C# code for parsing the files will be available, unless trivial (so basically for the print files, where not all lines will be identical). At a later stage I will possbily also have a binary format available - part of the data capture approach is in order to actually find out the possible range of values for some items I get text encoded so I can make a proper binary representation that is efficient - I plan on storing that complete data stream in a server for real time retrieval. The offer is one time. I have no intention with this in getting in as data provider. I just think it may be valuable for someone working on his own technology to have access to a high end stream to see what really goes on - and possibly stress test his technology. And other sources of that data are - hm - hard to find or hard to pay

Well, nice. I delayed the install for some days Happens Server R2 is out, and that has the ability to inject drivers during a Win7 install cycle (FINALLY) without the drivers being part of the WIM package. Time to upgrade my WDS first.

You could just realize that you are not the only one using a computer on the planet (which WOULD require you using your brain, incidentally) and that a lot of stuff is common sense and accepted good practices. Among them are not running crapstuff no important computers (i.e. isolating important applications on separate computers) and doing backups and having desaster recovery scenarios. Even many good trader books talk about it. For example... the OP obviously does not care about his money and would possibly gladly come back here whining about loosing funds when he is in a bad position and the power goes out. Others have a laptop with wireless (as in: UMTP / mobile phone based) internet as backup. Even others really do not care too much because their trading platform sits in a data center with so many redundant systems for power supply and internet connectivity that broken connections are really a rare incident. But still have backup plans. THis is not me playing smarts (it is you demonstrating ignorance though), it is common sense and globally accepted practices. For some trading is important and we use real money, so protecting this money is - important. The OP obviously falls into the category of "lets ignore common sense and all the warnings about computers being unreliable and then go on and whine about how bad the world treats me".

And then come whining when something goes back and someone is forced to use the common sense approach and making a backup? Yeah. Maybe the poster is a masochist and likes a bad system for trading. Definitely a reason to use the approach he does.

What else IS there on the computer? Seriously. Computer are cheap. I sit in front of two - and one is reserved for trading. No TS here (I currently use NInja), but there is NO Reason to have anything else on a trading system. If you can not afford 2 computers (whow) or a replacable hard drive (good enough - just swap out hard drives) then get a virtualization platform and trade from a virtual instance. There is NO excuse for running something as important and sensitive as trading shared with a ton of other software. Not said TS did not do something bad, but you should trade on a more segregated level. Tomorrow is my upgrade day, starting to wipe my computers and install the RTM version of Windows 7 Enterprise for trading

You can not. Multiple instruments is not supported at current NInja versions. It is one of the big new features coming next year with Nt7.

Pretty ridiculous from NeoTicker. Especially in regards to automated trading quite a lot of important questions are the "how". Like - what language. I take more than 5 minutes trying to find out the basics, it shows me it is not a worthy product. They do not care enough My plans for now are easy: * Go on direct trading with NinjaTrader Direct Edition (free from Mirus for my account). * Automate my trading and data collection using R Api to work against Zen-Fire. The whole thing may turn into an open source and/or closed source framework/product. Not sure. That being said: I am very gratefull to the people at Rithmic. Top notch support. Their API (c++ hardcore, so to say) has some rough edges, but they help me a lot. I am currently busy wrapping that API up in a Managed C++ library for my own use in a C# application

Actually no, I do not use Ninja with my own code. I use both, NT and my own code side by side on different accounts I work on getting NInja totally out. I STARTED by using the Zen-Fire API directly, but realized it takes a little longer than I want to be able to do all things, so I now run NinjaTrader on a separate account side by side. Great for manual order entry etc. I tried programming, but it did not work out. From the account number actually I realized that NInjaTrader is NOT using the Zen-Fire API - and by that time I actually already had access to the REAL api to the Zen-Fire system, which is basically the Rithmic API, which I am now putting into my code. I realized that, btw., bcause the account number that Ninja Trader gave me includes the clearing group, while I could not get the same out of the Zen-Fire API by any means - the API cuts the string, it seems. On top, NInja Trader does not have the Zen-Fire api dll anywhere Guess what - they wrap the Rithmic api themselves, too. My main complaint with Zen-Fire was historic information access. Not like "ok, i need the last 3 weeks" but as in "my internet was down 10 minutes, what fills exactly id i get when and what ticks happened". Zen-Fire has no API to get historical audit info or even 10minute historical quotes. The "real" zen fire API (i.e. bythe producers of the system - which is more widely used than you think, btw.) has. I can ask for all EXECUTIONS back, I can ask for tick data back (one day at the moment, but they plan to expand that). NeoTicker was not on my desk. TickZoom was (discarded), OpenTrader is. I do a lot of automatism stuff at the moment. Just looked at NeoTicker, and the website is as non-informative as possible.

Actually no. But if you use the REAL api, then you can do a LOT more than what the ZenFire API Exposes. THIS is what I blame them for. I actually use the ZenFire API myself. It - sorry - is as bad as the NInja API. Beginner mistakes partially - for example exception handling. If you drop it and use the REAL api (i.e. the one desigend for the technology - a C++ lib file, which incidentially is also what is used to build the official zen fire API), you get a lot more functionality: Examples: * Subscription to exchanges * Risk analysis information from the backend * Requests for limited historical data (same day), which still comes in handy when you have to reset routers etc. And in fact a lot more. Basically, Zen-Fire is RIthmic repackaged, and when writing the Zen-Fire API they "forgot" a lot of the nice features of the platform.

[Qute]Zenfire is an order execution and real time pricing engine, arguably one of the best[/Quote] Wrong. Totally wrong. Zen-Fire is a marketing name for a much more common platform that is developped and maintained by another company. Whoever uses Zen-Fire is buying a repackaged product In fact, Zen-Fire has a weaker API than the original product, because whoever made the API Decided to drop a significant part of the functionality. This is extremely obvious in NInja-Trader which is NOT USING THE ZEN-FIRE API. No joke - if you get the API and look at Ninja you find out they... do not use it. They use the "real deal". Rithmic has a great platform, in use by quite a lot more brokers. That Mirus and others decided to wrap it under a different name. From my experiences, they add pretty much nothing to it (as in: no additional functionality). So, technically, anyone using the Zen-Fire API is doing a stupid thing because he limits himself to ONE Rithmic system (because the Zen-Fire API has all the connections hardcoded), while using the real API he could connect quit a lot more different brokers

As was said, get another data feed. THat said, they are all limited. There is none offering you full historical feeds - which makes backtesting strategies a little pointless as you can not use filters based on level 2 quote "quality" (i.e. stay out if there is not enough depth, like around major news, when the market gets super-think in some instruments). People loving NInja are mostly not exactly good programmers. TradStation set the bar terribly low, and Ninja tries to stay that low - they seriously abuse the .NET programming model in order to allow a "function like" approach to defining indicators. This is maybe fine for people coming from TS etc., but I sadly earned my life as application architect. I see all the problems of their approach As I have personalyl never really used TS for extensive periods, no comment on that from me

Not sure about the complete order book (though it may come in handy at times - like to see when it is particularly shallow and then to stay out of the market- happens in the YM at times before news). But at least best bid and ask are conditio sine qua non in some markets. Try trading emini financial futures during exchange nights. The bid and ask move in perfect sync with forex quotes - obviously, enough arbitrage companeis there. Sadly, it rarely trades. It trades SO rarely, stops are useless.... one has to program ones own stop mechanism to react on bid/ask running through ones stop price. I had (in simulator) situations where bid and ask passed through my stop about 30 ticks before someone decided to trade one contract (which triggers the stop). Perfect in sync with exchange regulations (trades trigger stops, only), but... in this case the trades only are just not the real representation of the market.

In general, as note - all that oddities that you see will not go away, they become more. I disgustingly demanded a refund yesterday for my NT license given that the more serious stuff I do, the more I get demonstrated that - with all respect they deserve - whoever is in charge of quality control at Ninja should sell burgers at McDonalds and not work in IT anymore ever. The lack of daily data (which is terrific in the oversight) as well as a ton of idiocies in API and implementation is ridiculous. Granted, Ninja Trader is among the better platforms. More shocking than anything else. Btw., you all are aware that Zen-Fire is the "data feed that does not exist"? I mean, if you think Zen-Fire is a technology platform, you probably would also see me as car producer if I just change the manufacturer name on cars I sell? Zen-Fire is a simple wrapper around another provider's API and resells that under their own name. Their API is seriously limited compared to the real deal - probably the reason that Ninja itself does not use the official Zen-Fire API but the one provided by the real technology provider. There is nothing particular great thing about Zen-Fire that you can not get from half a dozen clearing firms if you know what to ask about If you do API programming, I can only sugggest you do the same- stay away from Zen-Fire and use their backend with the "real" api from their technology provider. Not only does this make you more independant, you also get a lot more and cleaner functionality (among them for example audit trails to ask for the history of fills should you experience a down time, and risk allowances of your account). The real API also has (limited) functionality to request historical data (same day only at the moment, but I was told they work on expanding that - not only to go 3 years back, but also considering covering bid/ask - which is tremendously important on some rarely traded instruments where bid and ask still move). If you only need daily bars, you can easily go with somehting like eoddata.com - pretty cheap compared to real time data providers. You may also check out google financial API's (free), even for intra day data (though not in extreme granularity, but a lot better than daily bars).

Stupid question, but what is the sense of that? THere are numerous APIs allowing access to hourly data rages (google financials for example). Unless one gets down to the roots and collects tick data including bid and ask it is useless for short term analysis and it is in particularly useless for trade simulation, especially during off times. At least in Futures, data providers allowing the capture of complete exchange data (CME, CBOT) are not that expensive.