[ZOSO]

Originally Posted by UrmaBlume »

Why?

Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money.

Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want.

Plus with all the great open source db's available why MS db?

For Historical data you can get a free trial for TS and download what ever you want.

Is the sole reason for this app to save $99 a month? I don't get it.

There you go again for the fourth time plugging TradeStation sans their ad.

Was it your intent to hijack this thread to talk up TS or was it an accident?

[jmayo05]

It sounds like you and I have very similar interestes. I would love to be involved in the project. Admittedly, I don't know a whole heap in application programming. (More knowledgable in web development and server hardware.)

I'm currently employeed at an FCM to monitor and administer an in-house trading system. My overall goal is to test system ideas and market trends that I notice while watching the futures markets at work. If you need resources, such as a server to host the data, or whatever, loop me in and I would love to help out.

[filip]

I'm SQL Server database administrator. If you need my help about sql, database, indexes,...etc let me know, I'd like to take part in this project. Actually I've developed something but it has manual export from my provider and then import and analyse.
F

Originally Posted by Szymon »

Hi All,

I'm starting this thread, it is going to be based on open source license.

The aim of this application project is to produce software that will record hourly price of instruments into a MS Sql server. This server can be sitting locally or somewhere else via the web, you can then use .NET xml technologies to retrieve information.

The first part of the project is to find out a cheap or free hourly market data provider that has an API feed.

What are the advantages of this kind of application? Well for all you system builders out there, You can then develop other applications that can utilize the information and build systems with your formulas and strategies that your third party applications cannot provide.

After we will find the right market data source. We will then get into developing the database structure. I will post SQL files here so you can create your own databases. At this stage I am only interested in 1hour time frame so that will be the focus for this project but you can change that to suit any other time frame.

The last part of the project is to develop using vb.net a small application that will utilize the market data API feed and record new bar information into the database. The information that I want to record is OPEN, CLOSE, HIGH and LOW for multiple instruments, with an update feature in case the comptuer has been switched off for a number of hours.

I am willing to provide a MS 2008 database that will have web access available but only to those people that will assist me in this project. I will build a .NET web query xml page where you can utilize this information for yourself and your other projects.

By having multiple clients talking to the database server, no matter what machine is switched off the database is constantly being updated from all those other machines that are switched on. By helping you have a choice to utilize this information for your own projects.

I hope this makes sense.

If you have any questions please do not hesitate to ask me.

Regards

Simon

[filip]

I'm SQL Server database administrator. If you need my help about sql, database, indexes,...etc let me know, I'd like to take part in this project. Actually I've developed something but it has manual export from my provider and then import and analyse.
F

[dam5h]

DISCLAIMER: i'm not very familiar with sql, though i've used it once or twice. so i'm just throwing this out there for anyone as an fyi for anyone interested in a similar project to this thread that make seek an alternative framework.

what i have used and really liked for historical backtesting is pytables, it's pretty awesome--can natively treat a portions (amount currently in RAM) of a database as a numpy array and has some real speedy and easy to use c based find/matching functions. i've used TA-Lib wrapped in python (but actually run in c) to populate the tables in the case that i wanted some indicators

plus, its python ; so we talking rapid development times.

matplotlib and scipy have some cool charting capabilities including candlesticks in the finance module.

not sure if it can be used with ms .NET's iron python distribution, i've only used it with linux

PyTables - Getting the most *out* of your data
An Introduction to PyTables video tutorial - PyTables Tutorials - Learn: programming_tools, interface, extremely_large_datasets, pytables, HDF5, efficient, tutorial, tools, cover, data, solution, interfaces, free, videos, video, fast, features, infor

Numpy Home Page
matplotlib: python plotting — Matplotlib v0.98.6svn documentation
TA-Lib : Technical Analysis Library - Home

[marcj]

Generally, you sign legal agreements when you buy data, not to destribute it to others. So is this legal?

[AldPixto]

I've put several years of futures price data into MySQL tables. My thoughts are:

1 - Timezones are important
1a - The database should self-identify what timezone data is being stored in, e.g. some integer offset relative to UTZ (i.e. keep database in EST, CST, whatever makes sense to you)
1b - Data loading applications need to know what timezone the source data is in so they can normalize to the target database (becomes important when people in different timezones can contribute data)
1c - User Applications (e.g. charts) should be able to convert database times into end-user times

2 - Even if you're only computing or plotting "floor" session data, it is useful to have 24-hour data (e.g. globex) so that overnight high low prices are known

3 - Market Profile charts
3a - the maximum time quanta is 30 minute bars, the "current" bar can be updated with greater frequency (e.g. every 30 seconds) for semi-realtime charts
3b - Volume-at-Price data is necessary for Market Profile but it will force you to compromise the minimum price-delta (e.g. $1 for $BRK.B , $0.01 for $GLD)
3c - separate tables for OHLC (key: YYMMDD,HHMM) and Volume-at-Price (key: YYMMDD,HHMM,Price) help speed charting for MP and non-MP charts

4 - Store dates and prices as integer values
4a - need date methods to compute yymmdd+date_offset and (yymmdd+-yymmdd) as date offset (my database is in yymmdd and I will have to convert to yyyymmdd by next year)
4b - convert fractional prices (ags, bonds) to integers so that ranges can be easily computed, e.g. multiply Ag prices by 25 and 30-Year bonds by 15625 to eliminate the fractional portion of the price

5 - DBA stuff - data sources are imperfect and I have to reload data to remove bad ticks or fill-in missing times more often than I'd like. the real problem is finding bad data or missing ticks before the tick data goes offline (DTN tick data only goes back 8 days, eSignal tick data goes back 10 days)

[dam5h]

i really like the date/time module in python, it handles all that timezone, hour, and day of week stuff. not saying that you would, but if you ever wanted to do historical testing specific to days of week, you're covered without any tedious coding on your end.