[darthtrader]

Does anyone on here archive tick data into a database for future use?
With DTN they give you 8 days but it just seems like a waste to not store this information.
What I would love is a program that can store the info into a database that is open as possible so that it could be used no matter what software I change to in the future.
Anyone into this stuff? Maybe a straight out access database or sql db?

[mister ed]

Very good question Darth ... I will do some research and see if I can come with anything.

[Sparrow]

Darth, you could easily code that with NT, performance would get slightly worse because of having to use real time indicators. Either use market analyzer or 1 tick charts.
I'm doing a variation of this to export 1min price/volume data for a volume based price/volume histogram. But have been to lazy to code the histogram so far .

Edit: I complicated things a bit, NT can export tick data just fine, but I think it will only do some if it's already in the DB.

[Sparrow]

MySQL, or PostgreSQL would be an option for storing such data. Maybe also SQL Server Express or Compact but no idea if they have got space constraints or are really suitable for the job.
I'd go with MySQL because it's more optimized for speed and the stuff you'd like to do is fairly simple.

I suppose the tables could grow quite large but with today's hdds, it shouldn't be a problem.

If you get the data into some kind of csv format, you'd be able to load the data like this into a MySQL db.

I think it would be tedious work if you cannot automate the export/import. Probably there's a way to this but none of the limited amount of trading software I know allows command line export of data.

[darthtrader]

Hey guys,
I think Ninja can actually do this to some degree if you check "store realtime bar data" in options.
My interest though is to get into something more standard/open incase I'm not even using Ninja 3 years down the line. I guess in the meantime the market scanner in ninja would be the way to go.
I found some interesting post on elite searching for SQL. One thing that was interesting that I didn't know is that I guess relational databases aren't really ideal for time series data. It was mentioned a vector database is the way to go.
Someone posted a link to an open source vector DB and a commercial version
http://kx.com/
http://hdf.ncsa.uiuc.edu/HDF5/index.html

Could be interesting. MySQL would be cool but if I set this project up I don't want to hit a brick wall 3 years down the line because the DB is so huge and not using the right tool for the job.

[Sparrow]

Interesting Darth, however it's questionable if storing the data in a database is necessary at all. If you intend to generate statistical studies with secondary tools from the data it should be in a format that is widely used. The only thing you'd need to worry about is bringing it into the right format (5min/monthly/etc.)
In case you want to write your own studies, a database makes sense, otherwise text files are more efficient which you might choose to compress or not.

It could also be better to just purchase the data and save yourself all the hassle.

[darthtrader]

Ahh yea, I ran across that idea tonight too. That actually sounds like what the guys who are decent programmers were doing. Just throwing it in a binary/text file and reading in straight away.
Of course buying it would probly be alot better time spent. Nothing though that I really need right now, I would just like to learn this stuff to have a spare computer capturing data for the hell of it I guess.

[micro]

Originally Posted by darthtrader »

Does anyone on here archive tick data into a database for future use?
<snip>
Anyone into this stuff? Maybe a straight out access database or sql db?

Hi,

Check out http://tree.sourceforge.net. Datafeed is restricted to IB.

Regards,