Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

NetTecture

Anyone Interested in a Complete CME Feed Sample?

Recommended Posts

Hello folks ;)

 

As part of my development process for a trading platform, I am testing my data collection system next week. This is a stress test and will take 1-2 weeks - I can not rule out I have to restart of get a crash first week, and I Want a complete week sample.

 

Anyhow, I will collect a complete data feed from the CME, electronic trading (basically: the same you get via zen-fire, just COMPLETE, not filtered like zen-fire is, and not limited by symbol).

 

This will include:

* Preopening, openeing, closing etc. information

* Trades, obviously

* Best bid, Best ask, Bids AND Asks, including their invalidation (setting the bid / ask volume to 0 when the price moves out and the exchange does not track them.

 

This will include the following instruments:

* Naturally all futures.

* All options, are published by the exchange. This is the majority of data.

* Virtual instruments, such as spreads trades that are done on the exchange level. Most people won't know that but you can execute those on the exchange, which guarantees the spread value then. Those CAN be ignored - spread executions also show up as the separate leg trades in the instruments.

 

The timestamps are in microsecond format from the original data capture equipment. The stream is totally unedited. It does NOT include cancels an corrections to my information. I record it to get an idea of the volume, analyse that my data capture approach and get an idea how to actually process that hugh amount of data - as a stress test for my equipment.

 

Anyone interested in a copy (ONLY for the complete data) can contact me via PM. This is initially a free offer, but if the volume goes too high up I may have to ask for a small contribution for the data transfer amounts. Can not have 20 people downloading 50gb or so for free ;) We talk of SMALL here - basically covery my traffic costs. Not sure how large the archive will be - ask me in 2 weeks. Anyone else is free to put that archive up on peer to peer - I even may do so, but with a very limited bandwidth ;)

 

The file format will be tab separated windows text file (CRLF delimiters), naturally compresed, most likely with 7zip ;) Expect SIGNIFICANT Archives. We talk of 30.000+ lines PER SECOND in the the hot phases. Even during the night, often, the lines are not readable on a text output, because every trade results in a lot of bid adjustments. It will contain two file sets - one with the exchange prints, one with instrument metadata (name, allowed prices etc.). I may actually have to split the allowed prices out into a third file for technical reasons.

C# code for parsing the files will be available, unless trivial (so basically for the print files, where not all lines will be identical).

At a later stage I will possbily also have a binary format available - part of the data capture approach is in order to actually find out the possible range of values for some items I get text encoded so I can make a proper binary representation that is efficient - I plan on storing that complete data stream in a server for real time retrieval.

 

The offer is one time. I have no intention with this in getting in as data provider. I just think it may be valuable for someone working on his own technology to have access to a high end stream to see what really goes on - and possibly stress test his technology. And other sources of that data are - hm - hard to find or hard to pay ;)

Share this post


Link to post
Share on other sites

Ok, status update ;)

 

Change of hearts. Well, not really.

 

* Data collector running since yesterday around 1900 UTC (for those now knowledgable: That is Greenwhich Mean Time WITHOUT summer/winter time - computers use that internally).

* The log file so far has the form like

 

009-08-24 04:26:24.149830 ZDU9 CME Bid 0 9534

009-08-24 04:26:24.149830 ZDU9 CME Bid 10 9533

009-08-24 04:26:24.150741 DDU9 CME Ask 2 9572

009-08-24 04:26:24.150741 DDU9 CME Bid 0 9530

009-08-24 04:26:24.150741 DDU9 CME Bid 0 9547

009-08-24 04:26:24.150741 DDU9 CME Bid 2 9546

009-08-24 04:26:24.154835 ZDU9 CME Bid 2 9539

009-08-24 04:26:24.154835 ZDU9 CME Bid 12 9538

009-08-24 04:26:24.154983 DDU9 CME Bid 3 9529

009-08-24 04:26:24.155119 YMU9 CME Bid 4 9546

009-08-24 04:26:24.155385 ZDU9 CME Bid 0 9547

009-08-24 04:26:24.155385 ZDU9 CME Bid 1 9528

009-08-24 04:26:24.157930 DDU9 CME Bid 0 9548

009-08-24 04:26:24.157930 DDU9 CME Bid 2 9547

009-08-24 04:26:24.157930 DDU9 CME BestBid 2 9547

009-08-24 04:26:24.158331 YMZ9 CME Bid 2 9487

009-08-24 04:26:24.158331 YMZ9 CME Bid 3 9486

009-08-24 04:26:24.158331 YMZ9 CME Bid 0 9384

009-08-24 04:26:24.158331 YMZ9 CME Bid 1 9487

009-08-24 04:26:24.158331 YMZ9 CME BestBid 1 9487

009-08-24 04:26:24.158483 ZDU9 CME Bid 0 9548

009-08-24 04:26:24.158483 ZDU9 CME Bid 2 9547

009-08-24 04:26:24.164320 DDU9 CME Bid 0 9529

009-08-24 04:26:24.172657 ESU9 CME Bid 91 1030.75

009-08-24 04:26:24.172657 ESU9 CME Bid 100 1030.5

009-08-24 04:26:24.175702 ESU9 CME Bid 76 1029.75

009-08-24 04:26:24.175702 ESU9 CME Bid 128 1029.5

009-08-24 04:26:24.179895 YMZ9 CME Bid 0 9482

009-08-24 04:26:24.179895 YMZ9 CME Bid 10 9480

009-08-24 04:26:24.183040 ZDH0 CME Bid 0 9389

009-08-24 04:26:24.183040 ZDH0 CME Bid 5 9386

009-08-24 04:26:24.183040 ZDH0 CME BestBid 5 9386

009-08-24 04:26:24.199314 ZDZ9 CME Bid 0 9483

009-08-24 04:26:24.199314 ZDZ9 CME Bid 1 9481

009-08-24 04:26:24.199314 ZDZ9 CME BestBid 1 9481

009-08-24 04:26:24.200362 6JU9 CME Ask 21 0.01055

009-08-24 04:26:24.200362 6JU9 CME BestAsk 21 0.01055

009-08-24 04:26:24.200463 6JU9 CME Bid 18 0.010548

009-08-24 04:26:24.200463 6JU9 CME BestBid 18 0.010548

009-08-24 04:26:24.220230 6JU9 CME Ask 1 0.010549

009-08-24 04:26:24.220230 6JU9 CME Ask 0 0.010554

009-08-24 04:26:24.220230 6JU9 CME Ask 83 0.010552

009-08-24 04:26:24.220230 6JU9 CME BestAsk 1 0.010549

 

It is slightly bad: the hour is in 12 hour format instead 24. Not too bad for my purposes though. Main problem, though: it is already around 800mmb big ;) I wont have the space for a whole week on the particular drive.

 

I will stop collecting at the end of the day when the ES has closed or tomorrow - depending on disc usage today. I will then finish creating the other files needed tomorrow (mostly the descriptions and pricing steps) and make the archive available.

 

I THEN go on with a binary representation, based most likely on delta storage (after all, things do not change that much, so I can but the price in one byte easily most of the time, between prints in one symbol). The idea is to store information "per symbol" in time slices of maybe 5 minutes (or more, depends on the amount of data I reallly get) in binary fields ;). And will see what loading that into my SQL Server says... (which has a lot more space than the drive set aside NOW for the log - I seriously did not expect THAT much data). The textual log file is simply waaaaayyyy tooo large. I will possibly reduce the granularity on the timestamp to about MS resolution. May be less. I do not really see a need for a better granularity than about 25ms (which is what NxCore uses, too).

 

I will most likely start collecting again next week (I actually want to make full collection for my own purposes starting 1st of September), but I may filter the data - I simply do not need data on virtual instruments (suchj as spreads) And have not exactly a large need for options, so I may filter out the order book there, keeping only best bid and ask ;)

 

For those interested, btw: CPU utilization on my collecting station is really low so far, and network IO is around 1.6 megabyte per minute. That is around 200kbit ;) I Post another update after market open - that is when it gets interesting.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • also ... and barely on topic... Winners (always*) overpay. Buying the dips is a subscription to the belief that winners win by underpaying - when in actuality winners (inevitably/always*) win by overpaying... it’s amazing the percentage of traders who think winners win by underpaying ... “Winners (always*) overpay.” ...  One way to implement this ‘belief’ is to only reenter when prices have emphatically resumed the 'trend' .   (Fwiw, While “Winners (always*) overpay.” holds true in most endeavors (relationships, business, sports, etc...) - “Winners (always*) overpay.”  is especially true for auctions... continuous auctions included.)
    • re:  "Does it make sense to always buy the dips?  “Buy the dip.”  You hear this all the time in crypto investing trading speculation gambling. [zdo taking some liberties] It refers, of course, to buying more bitcoin (or digital assets) when they go down in price: when the price “dips.” Some people brag about “buying the dip," showing they know better than the crowd. Others “buy the dip” as an investment strategy: they’re getting a bargain. The problem is, buying the dip is a fallacy. You can’t buy the dip, because you can't see the total dip until much later. First, I’ll explain this in a way that will make it simple and obvious to you; then I’ll show you a better way of investing. You Only Know the Dip in Hindsight When people talk about “buying the dip,” what they’re really saying is, “I bought when the price was going down.” " ... example of a dip ... 
    • Date: 19th April 2024. Weekly Commodity Market Update: Oil Prices Correct and Supply Concerns Persist.   The ongoing developments in the Middle East sparked a wave of risk aversion and fueled supply concerns and investors headed for safety. Hopes for imminent rate cuts from the Federal Reserve diminish while attention is now turning towards the demand outlook. The Gold price hit a high of $2417.89 per ounce overnight. Sentiment has already calmed down again and bullion is trading at $2376.50 per ounce as haven flows ease. Oil prices initially moved higher as concern over escalating tensions with the WTI contract hit a session high of $85.508 per barrel overnight, before correcting to currently $81.45 per barrel. Oil Prices Under Pressure Amid Middle East Tensions Last week, commodity indexes showed little movement, with Oil prices undergoing a slight correction. Meanwhile, Gold reached yet another record high, mirroring the upward trend in cocoa prices. Once again today, USOil prices experienced a correction and has remained under pressure, retesting the 50-day EMA at $81.00 as we moving into the weekend. Hence, despite the Israel’s retaliatory strike on Iran, sentiments stabilized following reports suggesting a measured response aimed at avoiding further escalation. Brent crude futures witnessed a more than 4% leap, driven by concerns over potential disruptions to oil supplies in the Middle East, only to subsequently erase all gains. Similarly with USOIL, UKOIL hovers just below $87 per barrel, marginally below Thursday’s closing figures. Nevertheless, volatility is expected to continue in the market as several potential risks loom:   Disruption to the Strait of Hormuz: The possibility of Iran disrupting navigation through the vital shipping lane, is still in play. The Strait of Hormuz serves as the Persian Gulf’s primary route to international waters, with approximately 21 million barrels of oil passing through daily. Recent events, including Iran’s seizure of an Israel-linked container ship, underscore the geopolitical sensitivity of the region. Tougher Sanctions on Iran: Analysts speculate that the US may impose stricter sanctions on Iranian oil exports or intensify enforcement of existing restrictions. With global oil consumption reaching 102 million barrels per day, Iran’s production of 3.3 million barrels remains significant. Recent actions targeting Venezuelan oil highlight the potential for increased pressure on Iranian exports. OPEC Output Increases: Despite the desire for higher prices, OPEC members such as Saudi Arabia and Russia have constrained output in recent years. However, sustained crude prices above $100 per barrel could prompt concerns about demand and incentivize increased production. The OPEC may opt to boost oil output should tensions escalate further and prices surge. Ukraine Conflict: Amidst the focus on the Middle East, markets overlooking Russia’s actions in Ukraine. Potential retaliatory strikes by Kyiv on Russian oil infrastructure could impact exports, adding further complexity to global oil markets.   Technical Analysis USOIL is marking one of the steepest weekly declines witnessed this year after a brief period of consolidation. The breach below the pivotal support level of 84.00, coupled with the descent below the mid of the 4-month upchannel, signals a possible shift in market sentiment towards a bearish trend reversal. Adding to the bearish outlook are indications such as the downward slope in the RSI. However, the asset still hold above the 50-day EMA which coincides also with the mid of last year’s downleg, with key support zone at $80.00-$81.00. If it breaks this support zone, the focus may shift towards the 200-day EMA and 38.2% Fib. level at $77.60-$79.00. Conversely, a rejection of the $81 level and an upside potential could see the price returning back to $84.00. A break of the latter could trigger the attention back to the December’s resistance, situated around $86.60. A breakthrough above this level could ignite a stronger rally towards the $89.20-$90.00 zone. Always trade with strict risk management. Your capital is the single most important aspect of your trading business. Please note that times displayed based on local time zone and are from time of writing this report. Click HERE to access the full HFM Economic calendar. Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding on how markets work. Click HERE to register for FREE! Click HERE to READ more Market news. Michalis Efthymiou Market Analyst HMarkets Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past perfrmance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in FX and CFDs products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • Date: 18th April 2024. Market News – Stock markets benefit from Dollar correction. Economic Indicators & Central Banks:   Technical buying, bargain hunting, and risk aversion helped Treasuries rally and unwind recent losses. Yields dropped from the recent 2024 highs. Asian stock markets strengthened, as the US Dollar corrected in the wake of comments from Japan’s currency chief Masato Kanda, who said G7 countries continue to stress that excessive swings and disorderly moves in the foreign exchange market were harmful for economies. US Stockpiles expanded to 10-month high. The data overshadowed the impact of geopolitical tensions in the Middle East as traders await Israel’s response to Iran’s unprecedented recent attack. President Joe Biden called for higher tariffs on imports of Chinese steel and aluminum.   Financial Markets Performance:   The USDIndex stumbled, falling to 105.66 at the end of the day from the intraday high of 106.48. It lost ground against most of its G10 peers. There wasn’t much on the calendar to provide new direction. USDJPY lows retesting the 154 bottom! NOT an intervention yet. BoJ/MoF USDJPY intervention happens when there is more than 100+ pip move in seconds, not 50 pips. USOIL slumped by 3% near $82, as US crude inventories rose by 2.7 million barrels last week, hitting the highest level since last June, while gauges of fuel demand declined. Gold strengthened as the dollar weakened and bullion is trading at $2378.44 per ounce. Market Trends:   Wall Street closed in the red after opening with small corrective gains. The NASDAQ underperformed, slumping -1.15%, with the S&P500 -0.58% lower, while the Dow lost -0.12. The Nikkei closed 0.2% higher, the Hang Seng gained more than 1. European and US futures are finding buyers. A gauge of global chip stocks and AI bellwether Nvidia Corp. have both fallen into a technical correction. The TMSC reported its first profit rise in a year, after strong AI demand revived growth at the world’s biggest contract chipmaker. The main chipmaker to Apple Inc. and Nvidia Corp. recorded a 9% rise in net income, beating estimates. Always trade with strict risk management. Your capital is the single most important aspect of your trading business. Please note that times displayed based on local time zone and are from time of writing this report. Click HERE to access the full HFM Economic calendar. Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding on how markets work. Click HERE to register for FREE! Click HERE to READ more Market news. Andria Pichidi Market Analyst HFMarkets Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in FX and CFDs products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • Date: 17th April 2024. Market News – Appetite for risk-taking remains weak. Economic Indicators & Central Banks:   Stocks, Treasury yields and US Dollar stay firmed. Fed Chair Powell added to the recent sell off. His slightly more hawkish tone further priced out chances for any imminent action and the timing of a cut was pushed out further. He suggested if higher inflation does persist, the Fed will hold rates steady “for as long as needed.” Implied Fed Fund: There remains no real chance for a move on May 1 and at their intraday highs the June implied funds rate future showed only 5 bps, while July reflected only 10 bps. And a full 25 bps was not priced in until November, with 38 bps in cuts seen for 2024. US & EU Economies Diverging: Lagarde says ECB is moving toward rate cuts – if there are no major shocks. UK March CPI inflation falls less than expected. Output price inflation has started to nudge higher, despite another decline in input prices. Together with yesterday’s higher than expected wage numbers, the data will add to the arguments of the hawks at the BoE, which remain very reluctant to contemplate rate cuts. Canada CPI rose 0.6% in March, double the 0.3% February increase BUT core eased. The doors are still open for a possible cut at the next BoC meeting on June 5. IMF revised up its global growth forecast for 2024 with inflation easing, in its new World Economic Outlook. This is consistent with a global soft landing, according to the report. Financial Markets Performance:   USDJPY also inched up to 154.67 on expectations the BoJ will remain accommodative and as the market challenges a perceived 155 red line for MoF intervention. USOIL prices slipped -0.15% to $84.20 per barrel. Gold rose 0.24% to $2389.11 per ounce, a new record closing high as geopolitical risks overshadowed the impacts of rising rates and the stronger dollar. Market Trends:   Wall Street waffled either side of unchanged on the day amid dimming rate cut potential, rising yields, and earnings. The major indexes closed mixed with the Dow up 0.17%, while the S&P500 and NASDAQ lost -0.21% and -0.12%, respectively. Asian stock markets mostly corrected again, with Japanese bourses underperforming and the Nikkei down -1.3%. Mainland China bourses were a notable exception and the CSI 300 rallied 1.4%, but the MSCI Asia Pacific index came close to erasing the gains for this year. Always trade with strict risk management. Your capital is the single most important aspect of your trading business. Please note that times displayed based on local time zone and are from time of writing this report. Click HERE to access the full HFM Economic calendar. Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding on how markets work. Click HERE to register for FREE! Click HERE to READ more Market news. Andria Pichidi Market Analyst HFMarkets Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in FX and CFDs products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.vvvvvvv
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.