Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

NetTecture

Anyone Interested in a Complete CME Feed Sample?

Recommended Posts

Hello folks ;)

 

As part of my development process for a trading platform, I am testing my data collection system next week. This is a stress test and will take 1-2 weeks - I can not rule out I have to restart of get a crash first week, and I Want a complete week sample.

 

Anyhow, I will collect a complete data feed from the CME, electronic trading (basically: the same you get via zen-fire, just COMPLETE, not filtered like zen-fire is, and not limited by symbol).

 

This will include:

* Preopening, openeing, closing etc. information

* Trades, obviously

* Best bid, Best ask, Bids AND Asks, including their invalidation (setting the bid / ask volume to 0 when the price moves out and the exchange does not track them.

 

This will include the following instruments:

* Naturally all futures.

* All options, are published by the exchange. This is the majority of data.

* Virtual instruments, such as spreads trades that are done on the exchange level. Most people won't know that but you can execute those on the exchange, which guarantees the spread value then. Those CAN be ignored - spread executions also show up as the separate leg trades in the instruments.

 

The timestamps are in microsecond format from the original data capture equipment. The stream is totally unedited. It does NOT include cancels an corrections to my information. I record it to get an idea of the volume, analyse that my data capture approach and get an idea how to actually process that hugh amount of data - as a stress test for my equipment.

 

Anyone interested in a copy (ONLY for the complete data) can contact me via PM. This is initially a free offer, but if the volume goes too high up I may have to ask for a small contribution for the data transfer amounts. Can not have 20 people downloading 50gb or so for free ;) We talk of SMALL here - basically covery my traffic costs. Not sure how large the archive will be - ask me in 2 weeks. Anyone else is free to put that archive up on peer to peer - I even may do so, but with a very limited bandwidth ;)

 

The file format will be tab separated windows text file (CRLF delimiters), naturally compresed, most likely with 7zip ;) Expect SIGNIFICANT Archives. We talk of 30.000+ lines PER SECOND in the the hot phases. Even during the night, often, the lines are not readable on a text output, because every trade results in a lot of bid adjustments. It will contain two file sets - one with the exchange prints, one with instrument metadata (name, allowed prices etc.). I may actually have to split the allowed prices out into a third file for technical reasons.

C# code for parsing the files will be available, unless trivial (so basically for the print files, where not all lines will be identical).

At a later stage I will possbily also have a binary format available - part of the data capture approach is in order to actually find out the possible range of values for some items I get text encoded so I can make a proper binary representation that is efficient - I plan on storing that complete data stream in a server for real time retrieval.

 

The offer is one time. I have no intention with this in getting in as data provider. I just think it may be valuable for someone working on his own technology to have access to a high end stream to see what really goes on - and possibly stress test his technology. And other sources of that data are - hm - hard to find or hard to pay ;)

Share this post


Link to post
Share on other sites

Ok, status update ;)

 

Change of hearts. Well, not really.

 

* Data collector running since yesterday around 1900 UTC (for those now knowledgable: That is Greenwhich Mean Time WITHOUT summer/winter time - computers use that internally).

* The log file so far has the form like

 

009-08-24 04:26:24.149830 ZDU9 CME Bid 0 9534

009-08-24 04:26:24.149830 ZDU9 CME Bid 10 9533

009-08-24 04:26:24.150741 DDU9 CME Ask 2 9572

009-08-24 04:26:24.150741 DDU9 CME Bid 0 9530

009-08-24 04:26:24.150741 DDU9 CME Bid 0 9547

009-08-24 04:26:24.150741 DDU9 CME Bid 2 9546

009-08-24 04:26:24.154835 ZDU9 CME Bid 2 9539

009-08-24 04:26:24.154835 ZDU9 CME Bid 12 9538

009-08-24 04:26:24.154983 DDU9 CME Bid 3 9529

009-08-24 04:26:24.155119 YMU9 CME Bid 4 9546

009-08-24 04:26:24.155385 ZDU9 CME Bid 0 9547

009-08-24 04:26:24.155385 ZDU9 CME Bid 1 9528

009-08-24 04:26:24.157930 DDU9 CME Bid 0 9548

009-08-24 04:26:24.157930 DDU9 CME Bid 2 9547

009-08-24 04:26:24.157930 DDU9 CME BestBid 2 9547

009-08-24 04:26:24.158331 YMZ9 CME Bid 2 9487

009-08-24 04:26:24.158331 YMZ9 CME Bid 3 9486

009-08-24 04:26:24.158331 YMZ9 CME Bid 0 9384

009-08-24 04:26:24.158331 YMZ9 CME Bid 1 9487

009-08-24 04:26:24.158331 YMZ9 CME BestBid 1 9487

009-08-24 04:26:24.158483 ZDU9 CME Bid 0 9548

009-08-24 04:26:24.158483 ZDU9 CME Bid 2 9547

009-08-24 04:26:24.164320 DDU9 CME Bid 0 9529

009-08-24 04:26:24.172657 ESU9 CME Bid 91 1030.75

009-08-24 04:26:24.172657 ESU9 CME Bid 100 1030.5

009-08-24 04:26:24.175702 ESU9 CME Bid 76 1029.75

009-08-24 04:26:24.175702 ESU9 CME Bid 128 1029.5

009-08-24 04:26:24.179895 YMZ9 CME Bid 0 9482

009-08-24 04:26:24.179895 YMZ9 CME Bid 10 9480

009-08-24 04:26:24.183040 ZDH0 CME Bid 0 9389

009-08-24 04:26:24.183040 ZDH0 CME Bid 5 9386

009-08-24 04:26:24.183040 ZDH0 CME BestBid 5 9386

009-08-24 04:26:24.199314 ZDZ9 CME Bid 0 9483

009-08-24 04:26:24.199314 ZDZ9 CME Bid 1 9481

009-08-24 04:26:24.199314 ZDZ9 CME BestBid 1 9481

009-08-24 04:26:24.200362 6JU9 CME Ask 21 0.01055

009-08-24 04:26:24.200362 6JU9 CME BestAsk 21 0.01055

009-08-24 04:26:24.200463 6JU9 CME Bid 18 0.010548

009-08-24 04:26:24.200463 6JU9 CME BestBid 18 0.010548

009-08-24 04:26:24.220230 6JU9 CME Ask 1 0.010549

009-08-24 04:26:24.220230 6JU9 CME Ask 0 0.010554

009-08-24 04:26:24.220230 6JU9 CME Ask 83 0.010552

009-08-24 04:26:24.220230 6JU9 CME BestAsk 1 0.010549

 

It is slightly bad: the hour is in 12 hour format instead 24. Not too bad for my purposes though. Main problem, though: it is already around 800mmb big ;) I wont have the space for a whole week on the particular drive.

 

I will stop collecting at the end of the day when the ES has closed or tomorrow - depending on disc usage today. I will then finish creating the other files needed tomorrow (mostly the descriptions and pricing steps) and make the archive available.

 

I THEN go on with a binary representation, based most likely on delta storage (after all, things do not change that much, so I can but the price in one byte easily most of the time, between prints in one symbol). The idea is to store information "per symbol" in time slices of maybe 5 minutes (or more, depends on the amount of data I reallly get) in binary fields ;). And will see what loading that into my SQL Server says... (which has a lot more space than the drive set aside NOW for the log - I seriously did not expect THAT much data). The textual log file is simply waaaaayyyy tooo large. I will possibly reduce the granularity on the timestamp to about MS resolution. May be less. I do not really see a need for a better granularity than about 25ms (which is what NxCore uses, too).

 

I will most likely start collecting again next week (I actually want to make full collection for my own purposes starting 1st of September), but I may filter the data - I simply do not need data on virtual instruments (suchj as spreads) And have not exactly a large need for options, so I may filter out the order book there, keeping only best bid and ask ;)

 

For those interested, btw: CPU utilization on my collecting station is really low so far, and network IO is around 1.6 megabyte per minute. That is around 200kbit ;) I Post another update after market open - that is when it gets interesting.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • Date: 11th July 2025.   Demand For Gold Rises As Trump Announces Tariffs!   Gold prices rose significantly throughout the week as investors took advantage of the 2.50% lower entry level. Investors also return to the safe-haven asset as the US trade policy continues to escalate. As a result, investors are taking a more dovish tone. The ‘risk-off’ appetite is also something which can be seen within the stock market. The NASDAQ on Thursday took a 0.90% dive within only 30 minutes.   Trade Tensions Escalate President Trump has been teasing with new tariffs throughout the week. However, the tariffs were confirmed on Thursday. A 35% tariff on Canadian imports starting August 1st, along with 50% tariffs on copper and goods from Brazil. Some experts are advising that Brazil has been specifically targeted due to its association with the BRICS.   However, the President has not directly associated the tariffs with BRICS yet. According to President Trump, Brazil is targeting US technology companies and carrying out a ‘witch hunt’against former Brazilian President Jair Bolsonaro, a close ally who is currently facing prosecution for allegedly attempting to overturn the 2022 Brazilian election.   Although Brazil is one of the largest and fastest-growing economies in the Americas, it is not the main concern for investors. Investors are more concerned about Tariffs on Canada. The White House said it will impose a 35% tariff on Canadian imports, effective August 1st, raised from the earlier 25% rate. This covers most goods, with exceptions under USMCA and exemptions for Canadian companies producing within the US.   It is also vital for investors to note that Canada is among the US;’s top 3 trading partners. The increase was justified by Trump citing issues like the trade deficit, Canada’s handling of fentanyl trafficking, and perceived unfair trade practices.   The President is also threatening new measures against the EU. These moves caused US and European stock futures to fall nearly 1%, while the Dollar rose and commodity prices saw small gains. However, the main benefactor was Silver and Gold, which are the two best-performing metals of the day.   How Will The Fed Impact Gold? The FOMC indicated that the number of members warming up to the idea of interest rate cuts is increasing. If the Fed takes a dovish tone, the price of Gold may further rise. In the meantime, the President pushing for a 3% rate cut sparked talk of a more dovish Fed nominee next year and raised worries about future inflation.   Meanwhile, jobless claims dropped for the fourth straight week, coming in better than expected and supporting the view that the labour market remains strong after last week’s solid payroll report. Markets still expect two rate cuts this year, but rate futures show most investors see no change at the next Fed meeting. Gold is expected to finish the week mostly flat.       Gold 15-Minute Chart     If the price of Gold increases above $3,337.50, buy signals are likely to materialise again. However, the price is currently retracing, meaning traders are likely to wait for regained momentum before entering further buy trades. According to HSBC, they expect an average price of $3,215 in 2025 (up from $3,015) and $3,125 in 2026, with projections showing a volatile range between $3,100 and $3,600   Key Takeaway Points: Gold Rises on Safe-Haven Demand. Gold gained as investors reacted to rising trade tensions and market volatility. Canada Tariffs Spark Concern. A 35% tariff on Canadian imports drew attention due to Canada’s key trade role. Fed Dovish Shift Supports Gold. Growing expectations of rate cuts and Trump’s push for a 3% cut boosted the gold outlook. Gold Eyes Breakout Above $3,337.5. Price is consolidating; a move above $3,337.50 could trigger new buy signals. Always trade with strict risk management. Your capital is the single most important aspect of your trading business.   Please note that times displayed based on local time zone and are from time of writing this report.   Click HERE to access the full HFM Economic calendar.   Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding of how markets work. Click HERE to register for FREE!   Click HERE to READ more Market news.   Michalis Efthymiou HFMarkets   Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in Leveraged Products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • Back in the early 2000s, Netflix mailed DVDs to subscribers.   It wasn’t sexy—but it was smart. No late fees. No driving to Blockbuster.   People subscribed because they were lazy. Investors bought the stock because they realized everyone else is lazy too.   Those who saw the future in that red envelope? They could’ve caught a 10,000%+ move.   Another story…   Back in the mid-2000s, Amazon launched Prime.   It wasn’t flashy—but it was fast.   Free two-day shipping. No minimums. No hassle.   People subscribed because they were impatient. Investors bought the stock because they realized everyone hates waiting.   Those who saw the future in that speedy little yellow button? They could’ve caught another 10,000%+ move.   Finally…   Back in 2011, Bitcoin was trading under $10.   It wasn’t regulated—but it worked.   No bank. No middleman. Just wallet to wallet.   People used it to send money. Investors bought it because they saw the potential.   Those who saw something glimmering in that strange orange coin? They could’ve caught a 100,000%+ move.   The people who made those calls weren’t fortune tellers. They just noticed something simple before others did.   A better way. A quiet shift. A small edge. An asymmetric bet.   The red envelope fixed late fees. The yellow button fixed waiting. The orange coin gave billions a choice.   Of course, these types of gains are rare. And they happen only once in a blue moon. That’s exactly why it’s important to notice when the conditions start to look familiar.   Not after the move. Not once it's on CNBC. But in the quiet build-up— before the surface breaks.   Enter the Blue Button Please read more here: https://altucherconfidential.com/posts/netflix-amazon-bitcoin-blue  Profits from free accurate cryptos signals: https://www.predictmag.com/ 
    • What These Attacks Look Like There are several ways you could get hacked. And the threats compound by the day.   Here’s a quick rundown:   Phishing: Fake emails from your “bank.” Click the link, give your password—game over.   Ransomware: Malware that locks your files and demands crypto. Pay up, or it’s gone.   DDoS: Overwhelm a website with traffic until it crashes. Like 10,000 bots blocking the door. Often used by nations.   Man-in-the-Middle: Hackers intercept your messages on public WiFi and read or change them.   Social Engineering: Hackers pose as IT or drop infected USB drives labeled “Payroll.”   You don’t need to be “important” to be a target.   You just need to be online.   What You Can Do (Without Buying a Bunker) You don’t have to be tech-savvy.   You just need to stop being low-hanging fruit.   Here’s how:   Use a YubiKey (physical passkey device) or Authenticator app – Ditch text message 2FA. SIM swaps are real. Hackers often have people on the inside at telecom companies.   Use a password manager (with Yubikey) – One unique password per account. Stop using your dog’s name.   Update your devices – Those annoying updates patch real security holes. Use them.   Back up your files – If ransomware hits, you don’t want your important documents held hostage.   Avoid public WiFi for sensitive stuff – Or use a VPN.   Think before you click – Emails that feel “urgent” are often fake. Go to the websites manually for confirmation.   Consider Starlink in case the internet goes down – I think it’s time for me to make the leap. Don’t Panic. Prepare. (Then Invest.)   I spent an hour in that basement bar reading about cyberattacks—and watching real-world systems fall apart like dominos.   The internet going down used to be an inconvenience. Now, it’s a warning.   Cyberwar isn’t coming. It’s here.   And the next time your internet goes out, it might not just be your router.   Don’t panic. Prepare.   And maybe keep a backup plan in your back pocket. Like a local basement bar with good bourbon—and working WiFi.   As usual, we’re on the lookout for more opportunities in cybersecurity. Stay tuned.   Author: Chris Campbell (AltucherConfidential) Profits from free accurate cryptos signals: https://www.predictmag.com/   
    • DUMBSHELL:  re the automation of corruption ---  200,000 "Science Papers" in academic journal database PubMed may have been AI-generated with errors, hallucinations and false sourcing 
    • Does any crypto exchanges get banned in your country? How's about other as Bybit, Kraken, MEXC, OKX?
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.