Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

SNYP40A1

Tick Data Storage and Relay

Recommended Posts

I am currently logging tick data into binary files on one computer (Computer A). But I am looking for a database to store the data and furthermore, I want to be able to query Computer A to backfill my charting software on another computer, Computer B. After backfilling, I then want Computer A to relay all received ticks relevant to the instrument(s) being monitored by Computer B to be forwarded to Computer B. I know that it's not a good idea to relay data for a true automated HFT system. However, I am not doing HFT and that latency should be ok for now, but I'd like to keep it at a minimum. I am using Linux for both systems. Does anyone know of a good open-source database solution and method for relaying the ticks? Would master-slave database replication be the way to go? At this point, my database would be not much larger than a couple GBs, I could flush the database to binary files at the end of each week to keep it small if necessary.

Share this post


Link to post
Share on other sites

Hack the market blog on HDF5 is about the only good info ive found on tick db construction:

Hack the market billions and billions

Hack the market managing tick data with hdf5

Hack the market tick data & hdf5 (part 2)

 

From what I've found the biggest thing is how many instruments you want to be logging.

If you only want to store a few then go with one of the open source relational packages but keep in mind it probably wouldn't be to hard to max out performance with a non time series db if you start adding instruments down the line.

Trying to roll my own tick db from parts has been a really demoralizing experience to be honest. Its a pretty thin number of users so there isn't so much to go on. Retail is using commercial solutions from the charting software and then institutions are using ultra expensive time series solutions like KDB+..so you are really on your own being in the middle.

Share this post


Link to post
Share on other sites

Actually though, if you are ok with flushing to binary files weekly, have you considered not even bothering with a db? Its hard to understand what you would be gaining from a db really with that time frame, unless these are baby steps of a much larger idea.

If you search on elitetrader for "tick database" or "tick db" and go back a few years there are some interesting discussions...In retrospect those discussions boiled down to morons like me trying to figure out how to use HDF5, berkeley db...monetdb now although I think thats too new to have come up on elite a few years ago.

Then there are guys in those discussions who realized this was a waste of time and just went with flat binary files...Don't even want to think about how much analysis they have done vs the time I've spent on this stuff...

Maybe I'm just hard headed but pytables/HDF5 is my last stand then I'm just going with binary files until its a problem...

this discussion will give you all the leads to search on you want in this area:

Nuclear Phynance

Share this post


Link to post
Share on other sites

Nate has nailed it really, pretty much anything will do unless you are dealing with lots (100's or maybe even 1000's) of instruments. The key thing is to structure your code properly so all data base stuff is done through a couple of primitive routines. More sophisticated stuff uses those primitives. If you architect sensibly you should be able to change at a later stage in hours or days rather than days or weeks. Go with what you know or fancy learning about.

Share this post


Link to post
Share on other sites
Nate has nailed it really, pretty much anything will do unless you are dealing with lots (100's or maybe even 1000's) of instruments. The key thing is to structure your code properly so all data base stuff is done through a couple of primitive routines. More sophisticated stuff uses those primitives. If you architect sensibly you should be able to change at a later stage in hours or days rather than days or weeks. Go with what you know or fancy learning about.

 

Forums - How do you guys store tick data?

 

Threads like that are what keep me searching though...It still strikes me though this decision comes down to KDB is the obvious choice, HDF5 or berkley is next up to fudge a KDB type setup then flat files if you just don't want to bother....

It depends on a philosophy i soppose that you aren't going to out time series a single time series..

Edited by TLAdmin
competitor URL removed

Share this post


Link to post
Share on other sites

Thanks Nate and Blowfish, I appreciate the info. I actually posted a thread over at "that other place" and came to the conclusion that binary files are the absolute fastest way to store tick data. The more I thought about it, it's not that hard to write some code that will search among the binary files for the proper range that one is seeking. In fact, since the data will be stored in time order anyways, I don't see what value a database would add for what I am considering now. I can always go DB later if the need arises.

 

I actually had read all those articles before you posted. If I went with a DB, it would probably be HDF5. Berkley DB supports concurrency (the concurrent version, data store version does not support concurrency at all) through internal locking. Most databases might work that way, but I don't want to ever have the writer blocked for a reader. Most important function of my tick datalogger is to log data. I was also concerned about the possibility of database corruption with HDF5. Unless the hard drives starts to fail, you can't really corrupt a binary file. So I may revisit this topic later, but for now, simple binary files seem to be the way to go for my current purposes. In any case, I appreciate the info!

Share this post


Link to post
Share on other sites

Maybe flat binary files with 'tree' like pointers into them. So you might have an index of days that pointed at an index of minutes that point to an entry point in the flat file. So to load from N days back you simply look at days [N] minutes [zero] to get your entry point into the flat file. intuitively that always seemed like a decent way to approach it to me.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • How's about other crypto exchanges? Are all they banned in your country or only Binance?
    • Be careful who you blame.   I can tell you one thing for sure.   Effective traders don’t blame others when things start to go wrong.   You can hang onto your tendency to play the victim, or the martyr… but if you want to achieve in trading, you have to be prepared to take responsibility.   People assign reasons to outcomes, whether based on internal or external factors.   When traders face losses, it's common for them to blame bad luck, poor advice, or other external factors, rather than reflecting on their own personal attributes like arrogance, fear, or greed.   This is a challenging lesson to grasp in your trading journey, but one that holds immense value.   This is called attribution theory. Taking responsibility for your actions is the key to improving your trading skills. Pause and ask yourself - What role did I play in my financial decisions?   After all, you were the one who listened to that source, and decided to act on that trade based on the rumour. Attributing results solely to external circumstances is what is known as having an ‘external locus of control’.   It's a concept coined by psychologist Julian Rotter in 1954. A trader with an external locus of control might say, "I made a profit because the markets are currently favourable."   Instead, strive to develop an "internal locus of control" and take ownership of your actions.   Assume that all trading results are within your realm of responsibility and actively seek ways to improve your own behaviour.   This is the fastest route to enhancing your trading abilities. A trader with an internal locus of control might proudly state, "My equity curve is rising because I am a disciplined trader who faithfully follows my trading plan." Author: Louise Bedford Source: https://www.tradinggame.com.au/
    • SELF IMPROVEMENT.   The whole self-help industry began when Dale Carnegie published How to Win Friends and Influence People in 1936. Then came other classics like Think And Grow Rich by Napoleon Hill, Awaken the Giant Within by Tony Robbins toward the end of the century.   Today, teaching people how to improve themselves is a business. A pure ruthless business where some people sell utter bullshit.   There are broke Instagrammers and YouTubers with literally no solid background teaching men how to be attractive to women, how to begin a start-up, how to become successful — most of these guys speaking nothing more than hollow motivational words and cliche stuff. They waste your time. Some of these people who present themselves as hugely successful also give talks and write books.   There are so many books on financial advice, self-improvement, love, etc and some people actually try to read them. They are a waste of time, mostly.   When you start reading a dozen books on finance you realize that they all say the same stuff.   You are not going to live forever in the learning phase. Don't procrastinate by reading bull-shit or the same good knowledge in 10 books. What we ought to do is choose wisely.   Yes. A good book can change your life, given you do what it asks you to do.   All the books I have named up to now are worthy of reading. Tim Ferriss, Simon Sinek, Robert Greene — these guys are worthy of reading. These guys teach what others don't. Their books are unique and actually, come from relevant and successful people.   When Richard Branson writes a book about entrepreneurship, go read it. Every line in that book is said by one of the greatest entrepreneurs of our time.   When a Chinese millionaire( he claims to be) Youtuber who releases a video titled “Why reading books keeps you broke” and a year later another one “My recommendation of books for grand success” you should be wise to tell him to jump from Victoria Falls.   These self-improvement gurus sell you delusions.   They say they have those little tricks that only they know that if you use, everything in your life will be perfect. Those little tricks. We are just “making of a to-do-list before sleeping” away from becoming the next Bill Gates.   There are no little tricks.   There is no success-mantra.   Self-improvement is a trap for 99% of the people. You can't do that unless you are very, very strong.   If you are looking for easy ways, you will only keep wasting your time forgetting that your time on this planet is limited, as alive humans that is.   Also, I feel that people who claim to read like a book a day or promote it are idiots. You retain nothing. When you do read a good book, you read slow, sometimes a whole paragraph, again and again, dwelling on it, trying to internalize its knowledge. You try to understand. You think. It takes time.   It's better to read a good book 10 times than 1000 stupid ones.   So be choosy. Read from the guys who actually know something, not some wannabe ‘influencers’.   Edit: Think And Grow Rich was written as a result of a project assigned to Napoleon Hill by Andrew Carnegie(the 2nd richest man in recent history). He was asked to study the most successful people on the planet and document which characteristics made them great. He did extensive work in studying hundreds of the most successful people of that time. The result was that little book.   Nowadays some people just study Instagram algorithms and think of themselves as a Dale Carnegie or Anthony Robbins. By Nupur Nishant, Quora Profits from free accurate cryptos signals: https://www.predictmag.com/    
    • there is no avoiding loses to be honest, its just how the market is. you win some and hopefully more, but u do lose some. 
    • $CSCO Cisco Systems stock, nice top of range breakout, from Stocks to Watch at https://stockconsultant.com/?CSCOSEPN Septerna stock watch for a bottom breakout, good upside price gap
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.