Archiving Tick Data?

[darthtrader]

Does anyone on here archive tick data into a database for future use?
With DTN they give you 8 days but it just seems like a waste to not store this information.
What I would love is a program that can store the info into a database that is open as possible so that it could be used no matter what software I change to in the future.
Anyone into this stuff? Maybe a straight out access database or sql db?

[mister ed]

Very good question Darth ... I will do some research and see if I can come with anything.

[Sparrow]

Darth, you could easily code that with NT, performance would get slightly worse because of having to use real time indicators. Either use market analyzer or 1 tick charts.
I'm doing a variation of this to export 1min price/volume data for a volume based price/volume histogram. But have been to lazy to code the histogram so far .

Edit: I complicated things a bit, NT can export tick data just fine, but I think it will only do some if it's already in the DB.

[Sparrow]

MySQL, or PostgreSQL would be an option for storing such data. Maybe also SQL Server Express or Compact but no idea if they have got space constraints or are really suitable for the job.
I'd go with MySQL because it's more optimized for speed and the stuff you'd like to do is fairly simple.

I suppose the tables could grow quite large but with today's hdds, it shouldn't be a problem.

If you get the data into some kind of csv format, you'd be able to load the data like this into a MySQL db.

I think it would be tedious work if you cannot automate the export/import. Probably there's a way to this but none of the limited amount of trading software I know allows command line export of data.

[darthtrader]

Hey guys,
I think Ninja can actually do this to some degree if you check "store realtime bar data" in options.
My interest though is to get into something more standard/open incase I'm not even using Ninja 3 years down the line. I guess in the meantime the market scanner in ninja would be the way to go.
I found some interesting post on elite searching for SQL. One thing that was interesting that I didn't know is that I guess relational databases aren't really ideal for time series data. It was mentioned a vector database is the way to go.
Someone posted a link to an open source vector DB and a commercial version
http://kx.com/
http://hdf.ncsa.uiuc.edu/HDF5/index.html

Could be interesting. MySQL would be cool but if I set this project up I don't want to hit a brick wall 3 years down the line because the DB is so huge and not using the right tool for the job.

[Sparrow]

Interesting Darth, however it's questionable if storing the data in a database is necessary at all. If you intend to generate statistical studies with secondary tools from the data it should be in a format that is widely used. The only thing you'd need to worry about is bringing it into the right format (5min/monthly/etc.)
In case you want to write your own studies, a database makes sense, otherwise text files are more efficient which you might choose to compress or not.

It could also be better to just purchase the data and save yourself all the hassle.

[darthtrader]

Ahh yea, I ran across that idea tonight too. That actually sounds like what the guys who are decent programmers were doing. Just throwing it in a binary/text file and reading in straight away.
Of course buying it would probly be alot better time spent. Nothing though that I really need right now, I would just like to learn this stuff to have a spare computer capturing data for the hell of it I guess.

[micro]

	Quote:	darthtrader
	Does anyone on here archive tick data into a database for future use? <snip> Anyone into this stuff? Maybe a straight out access database or sql db?

Hi,

Check out http://tree.sourceforge.net. Datafeed is restricted to IB.

Regards,

[BlowFish]

I used tradestation 2000i to store tick data for the longest time. I still maintgain that data in multicharts though it is less manageable. There are several off the shelf packages that have pretty robust data management facilities. (Neoticker instantly springs to mind).

[darthtrader]

I might have to just buy Neoticker at some point. I've read its pretty hard to top as far as its database goes. Still the only issue is see with that is running into something where you want to change software down the road and have this massive database of tick data that can't be converted and would cost far too much to buy.

In the meantime I'm just going with flat csv files and will progress as needed. From what i've read about sql is that its just overkill on the overhead and not really ment for time series data.

This DB was mentioned on elite, its for storing massive amounts of scientific data and might be interesting, free and open source.
http://www.hdfgroup.org/products/index.html