[AgeKay]

I am looking for a data feed with an API that allows me to do order book analysis on Eurex. This requires a data feed with an API that allows one to enumerate every single order book update (including trades) in the order it occurred. An added plus would be exchange time stamps with high resolution (i.e. milliseconds or smaller).

To do this for Eurex, that means that the vendor either has to subscribe directly to the Enhanced Broadcast Solution (EBS) at Eurex [members only so unlikely that vendors subscribe to it] or to the CEF ultra+ Eurex feed from Deutsche Börse (this is basically just EBS + trade recovery) [for non-members such as vendors]. I could find only a handful of vendors on the website of Deutsche Börse that are connected to the CEF ultra+ Eurex feed (one of them is CQG).

Another important part is that the API must allow one to receive order book updates and trades in a synchronous manner so that the correct order of events can be ensured. Also no updates may be missed when the thread that the event handler is raised on is blocked (e.g. due to long processing) [i.e. the API instead queues those updates].

The only data vendor I've found so far that seems to meet both these requirements is CQG (waiting for confirmation from CQG). Please share your experience with their API if you have any.

Are there any other out there besides CQG that meet my requirements?

[gooni]

Zen-Fire should do this, or at least seems to based on my usage of the API - even though I never used it for the level 2 feed (order book).

Although I very strongly recommend that you don't block the API even if you don't want to process the events immediately (ie you need guaranteed synchronous processing). I used a thread to pull the events from the API and place them in a local memory queue (actually a Queue in C#) and then processed them from that queue. If the API itself does it, all good and well - I'm just saying that it's not a requirement and none that I know do it (most APIs are geared towards technologically advanced users, all of which have their own methods for synchronising threads and prefer to retain control over it).

Best part: Zen-Fire API access is free. Can't comment on quality of data though.

[AgeKay]

Thank you for your detailed response gooni. Zen-Fire is a very fast feed which is excellent for execution but based on Fulcrum's investigations its data not always complete.

Thank you also for the heads up for not blocking the API. I don't do that anyway and always queue all events to be processed on another thread. The reason for this being a requirement is to make sure that the API does not miss any data during fast markets because even if you immediately pull the data and queue it, you can miss data if the API does not queue unprocessed data. At least that was my experience with the X_Trader API because their feed is also optimized for execution like Zen-Fire. On the other hand, I never miss any data from IQFeed sockets API but it's not connected to the data feeds I mentioned above (it's connected to CEF Core from Deutsche Börse that delivers netted order book data).

[gooni]

No problems AgeKay, just wanted to be sure you understood the difference between the API blocking and your event thread queuing.

Regarding Zen-Fire, the suggestion from FulcrumTrader is that the order book updates might come out of sequence with regards to the trades themselves, causing a mild discrepancy that over time could build up when one is doing bid-ask cumulative delta work.

However, you need to ask yourself whether this affects you. If you look at, say, a Ninja DOM when connected to Zen-Fire, the frequency of updates is quite good. Is this guaranteed to be in the same order that is disseminated by the exchange? I'd be amazed if it was. But is it good enough for what you want to do, given that it's free and easy to program to?

As always, it might be more prudent trying what's free and easy before attempting to expensively solve a problem you may never have.

[AgeKay]

gooni, thanks again for your concern.

For the kind of analysis I want to do I actually do need a "perfect" feed with all order book updates that can be enumerated in the order they were received. I've done lots of research for this already and unfortunately haven't found anything cheap that meets my requirements. I am not surprised though considering that this is pretty high-end analysis that I want to do (think what algos do). I actually just got confirmation from CQG that they do meet my requirements and they cost about as much as X_Trader and their API is cheaper than TT's API.

I've already implemented the order book visualization which would show me "interesting" behaviour in the order book, unfortunately it wasn't as helpful even though it was connected to feeds that are considered "good" for trading because I was missing lots of data (I could actually see that things were not adding up). If I get the feed I am looking for, then I am literally not going to miss anything going on in the order book, which would enable me to see things that guys like UrmaBlume are looking for.

[BlowFish]

Originally Posted by gooni »

Zen-Fire should do this, or at least seems to based on my usage of the API - even though I never used it for the level 2 feed (order book).

Although I very strongly recommend that you don't block the API even if you don't want to process the events immediately (ie you need guaranteed synchronous processing). I used a thread to pull the events from the API and place them in a local memory queue (actually a Queue in C#) and then processed them from that queue. If the API itself does it, all good and well - I'm just saying that it's not a requirement and none that I know do it (most APIs are geared towards technologically advanced users, all of which have their own methods for synchronising threads and prefer to retain control over it).

Best part: Zen-Fire API access is free. Can't comment on quality of data though.

I just started with Zenfire and C#. I have ticks running into a grid last, BB & BA no 'level II' yet. I am kind of learning it all as I go along. I came to a similar conclusion.....have a single queue for ticks that raises events for interested methods. Zen is not thread safe but using a single queue and then multi threaded from there on seems like the right way to proceed. Time to read the multi thread chapters of 'CLR via C#'. I have to say it seems like a massively complex environment. One thing that occurs to me is that there is no way to know if a race condition can occur within the Zen API itself.

[AgeKay]

Originally Posted by BlowFish »

Time to read the multi thread chapters of 'CLR via C#'. I have to say it seems like a massively complex environment.

I find multi-threading one of the hardest things to do. Debugging a threading problem is nightmare but VS 2010 supposedly makes it much easier (still using VS 2008). The best tutorial of threading in C# is this. Try to keep it simple. Most of the time, all you need to do is to prevent a piece of code from being entered by two threads at the same time to ensure data you access is the same it was when you entered that piece of code. You can use the 'lock' keyword on any instance of a reference type to do that. All the other times, I just use a 'ConsumerThread' class I developed that consumes data on another thread.

Originally Posted by BlowFish »

One thing that occurs to me is that there is no way to know if a race condition can occur within the Zen API itself.

This can only happen if they raise events on different threads. If they always raise events on the same thread, then no race condition. Best to just ask them.

[gooni]

Originally Posted by AgeKay »

The best tutorial of threading in C# is this.

This is the tutorial I used when writing my Zen-Fire API connector (C#). Can't recommend it highly enough. If you'd like some code, PM me.

Since I come from a messaging/integration background, I naturally created a single 'hub' that would receive ticks from Zen-Fire, and I then distribute to interested 'tick sinks' (which are plugins). Some tick sinks are files, others are the console itself, others are trading programs etc. This saves having to rewrite sections of code and manage multiple connections.

By far the largest issue was reconnection after a connection failure. The API is meant to take care of it internally, but it simply doesn't always work. I had to resort to actually killing my daemon and restarting it. This works nicely

Note that Zen-Fire is definitely not thread-safe. All access to its events etc should be single threaded. This is especially important for GUI applications (which I try and avoid).