Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

taotree

Members
 View Profile  See their activity

  • Content Count

    72

  • Joined

  • Last visited

Posts posted by taotree

  1. Posted

    I don't know of any feeds which supply the entire order book. They said that they don't filter their market depth data.

     

    I assume that's all he meant by order book--market depth to the same level as the exchange gives. For eMinis that's 10 levels, for other products typically 5 or whatever.

  2. Posted

    Oh, this is fun. So...

     

    First of all, let me say that when I looked at the other data it took me less than a couple minutes to find data that didn't line up. So either it does happen all over the place, or I just got really lucky. Sorry... not rigorous check yet since I'm still just exploring.

     

    So, I took a look at sample data direct from the CME. I spent a good deal longer finding ones that matched up--maybe 10 minutes? not sure, but then I found this:

     

    From: Top of Book

     

    2010011509564006628780EES F100300900 201132252A M M100115

    2010011509564006628780EES F100300157 201132002B M M100115

    2010011509564006628790EES F100300038 201132002 100115

    2010011509564006628800EES F100300900 201132252A M M100115

    2010011509564006628800EES F100300120 201132002B M M100115

     

    What this is showing is:

    First: ask = 900, bid = 157

    Then a trade of 38

    Then: ask = 900, bid = 120

     

    Oops, we gained an extra one. So... there is at least one example of the CME's own data not lining up. So apparently the CME doesn't require them to match up.

     

    By the way... it's not that a 1 difference in the bid is going to make or break a strategy. It's that these discrepancies are making it difficult for me to assess the accuracy of a data feed.

     

    Another one:

     

    2010011509564006628980EES F100300870 201132252A M M100115

    2010011509564006628980EES F100300098 201132002B M M100115

    2010011509564006628990EES F100300001 201132002 100115

    2010011509564006629000EES F100300870 201132252A M M100115

    2010011509564006629000EES F100300096 201132002B M M100115

     

    And another:

     

    2010011509564006629250EES F100300083 201132002A M M100115

    2010011509564006629250EES F100301185 201131752B M M100115

    2010011509564006629260EES F100300001 201132002 100115

    2010011509564006629270EES F100300076 201132002A M M100115

    2010011509564006629270EES F100301183 201131752B M M100115

     

    Found those two quickly after the first... Okay... now time to ask CME directly about this and see if I can dig into what's going on.

     

    At this point, I'm going to assume then that probably the only possible way of assessing the accuracy of a data feed is to collect the data, purchase CME data for the same day, and run a check to ensure every tick was received in the same order and context.

  3. Posted

    By the way... It appears that you can get a direct internet feed to CME for $500/month but... does anyone know of a data distributor that just directly passes on CME data for less than $500/month? Then it would be easy to verify that we're getting the best data (can purchase some historical CME data directly and compare it to confirm). Comparing to data feeds that change the format and such is difficult because it requires manipulation and even interpretation.

  4. Posted

    I guess once you get down to collecting your own data and daily reviewing it is ultimately a final resort..... I think you will unfortunately find this everywhere. It all depends on what you require it for. I have taken the view I just want data that is close enough.... and then I will take any test with a grain of salt.

     

    I'm a little confused--I would expect historical data to be more accurate than real time data. I was collecting my own data for a while, but I now know that the feed I was using is inaccurate so... collecting data doesn't do me any good unless I know the data I'm getting is good.

     

    And as for "close enough"... that's what I'm struggling with. If this type of discrepancy "happens quite often" how do I assess the quality of data? If there are confusing things happening in even the best quality of data available (which I do not know if that's the case), how does one differentiate between the best, the "close enough" and the way off?

  5. Posted

    So... there have been various discussions about data feeds and I'd like to see if I can find a way to ensure I have a quality one. I've heard some recommendations but...

     

    Here's why I ask. Supposedly, I would expect that CQG data factory would be of high quality. This is historical data, so there are no questions about connection issues.

     

    I downloaded their sample and found this example:

     

    F.US.EPM08 20080104 1 0918 144375 B N N 28

    F.US.EPM08 20080104 1 0918 144375 B N N 25

    F.US.EPM08 20080104 1 0918 144400 A N N 34

    F.US.EPM08 20080104 1 0918 144375 T N N 9

    F.US.EPM08 20080104 1 0918 144375 T N N 5

    F.US.EPM08 20080104 1 0918 144375 T N N 6

    F.US.EPM08 20080104 1 0918 144325 B N N 8

    F.US.EPM08 20080104 1 0918 144375 A N N 19

    F.US.EPM08 20080104 1 0918 144325 B N N 5

     

    Bid was at 25, then there were trades of 20 total and then bid went to 8.

     

    I asked them if that was normal and they said "what you see is normal and happens quite often". But we have 3 trade events in a row without any bid updates, and when we do get a bid update, it doesn't add up.

     

    I guess I might be able to get data from CME directly.

    http://www.cmegroup.com/market-data/datamine-historical-data/

     

    It would be very interesting to see if they have the same issue of numbers not all adding up.

     

    What I'm getting at is... it seems incredibly difficult to find and confirm that data is accurate and I'm starting to wonder if it's possible at all.

  6. Posted

    The issues is that it uses UDP and so there is the potential to loose packets. This does not happen (or happens rarely) with good infrastructure. I am also not convinced by the test methodology and data set used as a benchmark, FT does not seem to want to discuss that. :)

     

    Anyway loosing the odd packet is a far lesser evil than not having things sequenced correctly (wherever that occurs). There's the thing you have 3 asynchronous event streams (bid ask and last) they need to get sequenced.

     

    I have looked at event data for zen-fire and can say that there are definitely things that don't add up. Something like bid is at 19, there's a trade at same bid price of 25 and next update is bid is at 43 at same price. Obviously, there should have been some updates in the middle in order for that to have happened but they didn't show up in the feed I received.

     

    Also, I can attest to data arriving out of sequence.

  7. Posted

    This sounds like a similar problem to how Tradestation & Multicharts implement insidebid insideask but in the feed implementation. Last is not properly synchronised with bid / ask and race conditions can occur.

     

    I think it's quite different. Explained below.

     

     

    I am not sure your solution will get round the issue unless DTN send all order book changes (properly sequenced with BB & BA changes and no aggregation) as well as trades? What if the BB or BA moves due to order cancellation in between 2 trades?

     

    Based on FulcrumTrader's statements, DTN.IQ does send all order book changes properly sequenced. Now... given the situation in question let's consider, what IS the proper sequence?

     

    A trade comes across that will take out all the rest of the bid. Is it correct to show the trade first, or the bid update first? And this goes for all the time--should the trade show up first, or the book change first? From a programming perspective... I would remove the bid before I processed the trade, otherwise there might be potential synchronization issues. I grab the amount off the bid to "hold" it while I process the order. Kind of like how someone puts an authorization on your credit card pre-sale and then charging it post-sale.

     

    Now, in the data feed, that low level of trade stuff does not need to be reflected, but... It seems quite reasonable to show book updates prior to transactions. It could go either way--so long as you are consistent and always do it one way or the other.

     

    I would be interested... in the low level data of dtn.iq--are book updates always given before transactions? So a trade of 10 at bid will be always immediately preceded by a reduction in the bid size by 10?

  8. Posted

    Investor RT offers "DTN Market Access" which is historical only data from DTN for $15/month.

     

    So... I wonder if there is a way to get this without paying the fee for Investor/RT?Any chance I could get this for Ninjatrader, I wonder... DTN says $60/month for basic service, how are you avoiding that?

  9. Posted

    This is the exact reason why TradeVec is setting up connectivity at this time with DTN.IQ feed for those who need clean BID/ASK data. Handling even uncoalesced broker supplied data feeds has challenges for those who need proper BID/ASK data.

     

    Someone (I think on this thread) said that you could get delayed data from dtn for only $15/month. I haven't been able to find this information on their web site, though. Does anyone know where this is stated? Especially if it's possible to get this data through some type of API or download?

  11. Posted

    I'm not sure if it's the ninja software that has problems, the zen-fire datafeed, or both.

     

    Search some posts by FulcrumTrader as he has mentioned that he has analyzed zen-fire data and found it does have issues, and apparently Ninjatrader also has some issues with keeping up with data properly sometimes.

  12. Posted

    From what I understand we only get the update message on each trade.

     

    Let's say first trade is at 150.05 on the bid and we've never been higher. Then the bid goes 150.06 but there are no trades. and now you get an update message with bid 150.07 ask 150.08 and a trade at 150.06. we wouldn't have 150.06 in our dictionary?

     

    If the bid/ask changes were sent independently of the trades then everything would make a lot more sense. Please forgive me if I'm missing something really simple. :)

     

    It depends on your platform. Ninjatrader for example, the bid/ask changes are sent independently of the trades.

     

    In many platforms where your indicator is only updated on trades and you call some API to get the inside bid/ask... in some cases (Tradestation, for example) that means you might want to forget about trying to assign bid/ask to trades because it's using snapshotted bid/ask data which are stale. In some platforms, that can be accurate as it sounds like Agekay mentioned he was using one like that in a previous post.

  13. Posted

    I understand your example and I understand the algorithm of your code, but I don't see how (or when) your code will know that 123.22 went bid.

     

    are you programming to their API directly? Is their API C#?

     

    It's tracking each price and whether it was most recently a bid or an ask. So... when 123.22 went bid, it would be tracked as bid. When 123.21 went bid, it's tracked as bid. 123.22 is still in the dictionary as bid. So if a trade comes across at 123.22, it will be assigned as bid because that's the most recent value for that price.

  14. Posted

    I'm not interested in doing rebate trading exactly, but I am exploring a strategy that involves adding liquidity only (limit orders only) and is rather susceptible to commissions so... it seems that if I could find an instrument that offered rebates, it would be very interesting.

     

    Doing some searching, I find most references to rebate trading talking about stocks. Are there any futures or other non-stock, easy to day trade, instruments that offer commission rebates for adding liquidity?

  15. Posted

    The X_Trader API tells you which side the trade was on, so there are no trades without BB or BA.

     

    This statement suffers from a non-sequitur falacy. The first (the API tells you) does not imply the second (no trades without BB/BA). All it means is that X_Trader or the exchange or wherever that data is coming from is using some type of algorithm for you. It would be rather interesting to dig into that and track down exactly what algorithm they're using.

  16. Posted

    Just one more thing as Columbo might say.

     

    Seems to me a great working solution would be R|thmic / Zenfire or TT for live data and IQFeed for historical. I have not got round to testing the completeness of IQFeeds historical data but there are couple of people here that swear by it.

     

    If you're going to pay for IQFeed, why use a different feed for real-time? Why not use it for both real-time and historical?

  17. Posted

    Well, they don't. At least in none of the data feeds I've worked with can you get sequential order book updates. Instead you get the state of entire order book at one point in time.

     

    Which data feeds have you worked with? Aside from the drop/sequence issues, Zenfire does give you sequential order book updates. And from what FulcrumTrader has said, I get the impression that DTN does give you that without the issues.

  18. Posted

    If you are using the Zenfire' s API consider also the well know "UDP" problem of the feed...which invalidates many timestamp.

    Anyway I think that the "correct" sequence (send by the Exchange) can be preserved only by a "Ticker Plant" design platform.

    Zenfire doesn' t offer this...

    :-(

     

    All that has nothing to do with my comment--it's all still valid. The only thing you pointed out is that Zenfire does not satisfy the requirement in my last statement.

     

    And to be accurate... Zenfire's out of order problem doesn't invalidate the timestamp--it's just passing that along from the exchange--it invalidates the sequence of arrival.

  19. Posted

    You need more accurate time stamps to determine the order of events. It's annoying to have events occur at the "same time" even though you know they didn't. This might not bother most people, but I have programmed some pretty amazing stuff that would improve with more accurate time stamps. So this is not theoretical stuff, I actually need to know the sequence of events.

     

    You don't have to know the timestamps at all to know the sequence of events. So long as the data arrives in your algorithm in the proper sequence, that's all you need.

     

    I use an id encoder algorithm:

     

    In my code, I keep track of the most recent timestamps. When a new event comes in, I check to see if that timestamp has already arrived, if so, I increment a counter and tack that on to create my id, otherwise I tack on a 0 value for the counter. That allows me to preserve the sequence precisely regardless of the timestamp precision, and every event has a unique id.

     

    Now, that assumes that the data flow from exchange through feed provider all preserve the sequence of events as they send it to you--but... no matter what you're always beholden to the accuracy of the data feed.

  20. Posted

    I don't want to beat a dead horse, but even if the spread is 2 ticks, the "mid-point" of a 2 tick spread is still either bid or ask. If it looks as if it traded at that "mid-point" without being bid or offered, it just means that as soon as someone bid or offered, someone sweeped the market so quickly that you couldn't see it. I just want people to understand how orders are matched. There are no magic trades at mid-points on exchanges.

     

    Yes. As I mentioned before, you're talking about the internals of how exchanges work, and I'm talking about the actual data that arrives at your trading platform and how to code an indicator to deal with it. As you say, the two don't always match.

  21. Posted

    Not really. You are talking about two different things here. Stale data happens. But how is a trade at the mid-point supposed to happen? You can't enter limit orders at half a tick on an exchange, can you?

     

    Are you assuming a spread of only one tick? In that case, you are correct. However, many instruments have spreads of much greater than one tick. Speaking in terms of ticks, if bid is at 0 and ask is at 2 then a trade at 1 is at the midpoint.

  22. Posted

    I don't know what you are trading, but on exchange traded securities it can only trade on the bid or ask. I don't see how your data feed might screw that up since a mid-point requires both bid and ask as information.

     

    There may be a difference between what is conceptually required by the market and exchange and what actually comes across the wire in the data feed. Any filtered data feed or snapshotted one (IB, Tradestation, TT, etc.) will definitely have this problem since you cannot get the actual last bid and ask--you can only get a snapshot value that may be stale data.

     

    I was using zenfire which is supposedly unfiltered, but there apparently have been cases of this issue with it as well. FulcrumTrader has confirmed that zenfire does drop data sometimes, so that may be why--so maybe on a high quality feed, this would not be an issue. But... I don't know.

  23. Posted

    That's because a transaction at the mid-point is impossible. It's either traded at the bid or ask.

     

    My understanding is that is not the case--especially since the data stream is not perfect. And I know I have seen transactions occur on prices other than the last bid or ask I received from the data feed.

  24. Posted

    Someone asked me about this code and when I looked at it I noticed that it is missing the case where a transaction == mp, ie. when a transaction is at the midpoint between best bid and ask. I have read that at least one approach to that is to use whatever the direction was for the previous transaction for that case. Anyway... one would probably want to change the code to reflect that. Unless one wants to consider that if a transaction is exactly at the midpoint, then it's not directional at all and so ignoring it is what you want. Could argue either way I suppose.

     

     

     

    This is what I typically use in NT.

    // In vars section:
double lastAsk;
double lastBid;

// Methods
protected override void OnMarketData(MarketDataEventArgs e) 
{ 
 trackBidAsk(e);
 if (e.MarketDataType == MarketDataType.Last)
 {
   double mp = (lastBid + lastAsk) / 2;
   if (e.Price > mp)
   {
     // Transaction was at Ask, do whatever here.
   }
   else if (e.Price < mp)
   {
     // Transaction was at Bid, do whatever here.
   }
 }
}

 private void trackBidAsk(MarketDataEventArgs e)
 {
       if (e.MarketDataType == MarketDataType.Bid)
       {
	lastBid = e.Price;
}
       else if (e.MarketDataType == MarketDataType.Ask)
       {
	lastAsk = e.Price;
}
  }

  25. Posted

    Real simple.....DTN.IQ is matching CME data runs and I have NOW FOUND Zenfire/Rithmic feed runs that are NOT matching and that is a problem. I have ALSO since had verified what is causing the Rithmic/Zenfire feed data drops since I was myself using Zenfire feed with my NT charting (for back up and remote trading work).

     

    You mention you have verified what is causing the issue? Can you share that information?

     

    Any chance it's related to something I discovered while using the API... there are cases happening regularly where the data events are coming out of order. I grab the exchange timestamp to store data and since I use a counter just in case I get multiple items with the same timestamp, I discovered the out of order data right away and had to compensate for it. I haven't investigated to see what "damage" it might do to analysis, though.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.

  I accept