New Open Source Project - MS Sql Server Market Data Colletor.

[BlowFish]

Originally Posted by UrmaBlume Why? Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money. Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want. Plus with all the great open source db's available why MS db? For Historical data you can get a free trial for TS and download what ever you want. Is the sole reason for this app to save $99 a month? I don't get it.

I don't know what TS provide history wise but I would be surprised if it really does offer 'all the data you could want'. For example can you can load every tick of the ES since ist inception for example (complete with proper time stamps). What about the DAX FTSE CAC, what about more esoteric instruments? Data management is a pain but at least if you do that yourself you know you really do have everything you could possibly want andthat it has been collected and stored accurately.

I cant answer for Syzmone but why would you lock yourself in to an archaic propriatory product like TS? I presume the point of the project is so that you can connect your database to numerous data providers (including TS) and allow it to be queried by any client application that you can dream of.

Don't get me wrong TS is a great product for quick prototyping of ideas but as the cornerstone of a project to collect and manage financial time series data wel in fact TS as the cornerstone for any 'big iron' type project .....well no comment but good luck with that.

I do agree with your comment on the choice of database. SQL isn't ideal in the first place but despite that is still adequate. Going open would seem to make more sense to me too.

[Szymon]

Hi All,

Can someone tell me why MS Sql 2008 is not good for this, what have you experienced with this product and do you have any suggestions something else that is better than this product.

Here is a link to the MS Website for this product.

Microsoft SQL Server 2008: Overview


Simon

[uexkuell]

About ways to store large data sets please notice this extensive discussion (you may have to register first to get access to the yahoo-group):

Yahoo! Group TWS-API: Has anybody tried some sort of database?

It is a discussion group of people who are writing programs for IB (Interactive Brokers) API.
Anyway the discussion is relevant for everybody who thinks about saving (and accessing) a big amount of data.
Answers your question about SQL also.



Another thing that might deserve your interest:

Esper - Complex Event Processing

A free, open-source and working event-processing project.

[BlowFish]

One question to ask ...are SQL databases good the best way to store sequential data? Answer is probablly no. Does it matter that much? The answer is again probabbly not. Chances are the yahoo group that was linked will give you a much better idea of pros and cons.

Having chose SQl (which is certainly not wrong) then the only choice is which implementation to go with. To be honest I don't really know the current pros and cons, it's years and years since I have had to seriously evaluate DB products. Intuitivly an open source implementation seems to make sense for an open source project. There are many things that could blow that feeling right out of the water. Your preference is one of them

[ZOSO]

Originally Posted by UrmaBlume Why? Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money. Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want. Plus with all the great open source db's available why MS db? For Historical data you can get a free trial for TS and download what ever you want. Is the sole reason for this app to save $99 a month? I don't get it.

There you go again for the fourth time plugging TradeStation sans their ad.

Was it your intent to hijack this thread to talk up TS or was it an accident?

[jmayo05]

It sounds like you and I have very similar interestes. I would love to be involved in the project. Admittedly, I don't know a whole heap in application programming. (More knowledgable in web development and server hardware.)

I'm currently employeed at an FCM to monitor and administer an in-house trading system. My overall goal is to test system ideas and market trends that I notice while watching the futures markets at work. If you need resources, such as a server to host the data, or whatever, loop me in and I would love to help out.

[filip]

I'm SQL Server database administrator. If you need my help about sql, database, indexes,...etc let me know, I'd like to take part in this project. Actually I've developed something but it has manual export from my provider and then import and analyse.
F

Originally Posted by Szymon Hi All, I'm starting this thread, it is going to be based on open source license. The aim of this application project is to produce software that will record hourly price of instruments into a MS Sql server. This server can be sitting locally or somewhere else via the web, you can then use .NET xml technologies to retrieve information. The first part of the project is to find out a cheap or free hourly market data provider that has an API feed. What are the advantages of this kind of application? Well for all you system builders out there, You can then develop other applications that can utilize the information and build systems with your formulas and strategies that your third party applications cannot provide. After we will find the right market data source. We will then get into developing the database structure. I will post SQL files here so you can create your own databases. At this stage I am only interested in 1hour time frame so that will be the focus for this project but you can change that to suit any other time frame. The last part of the project is to develop using vb.net a small application that will utilize the market data API feed and record new bar information into the database. The information that I want to record is OPEN, CLOSE, HIGH and LOW for multiple instruments, with an update feature in case the comptuer has been switched off for a number of hours. I am willing to provide a MS 2008 database that will have web access available but only to those people that will assist me in this project. I will build a .NET web query xml page where you can utilize this information for yourself and your other projects. By having multiple clients talking to the database server, no matter what machine is switched off the database is constantly being updated from all those other machines that are switched on. By helping you have a choice to utilize this information for your own projects. I hope this makes sense. If you have any questions please do not hesitate to ask me. Regards Simon

[filip]

I'm SQL Server database administrator. If you need my help about sql, database, indexes,...etc let me know, I'd like to take part in this project. Actually I've developed something but it has manual export from my provider and then import and analyse.
F

[dam5h]

DISCLAIMER: i'm not very familiar with sql, though i've used it once or twice. so i'm just throwing this out there for anyone as an fyi for anyone interested in a similar project to this thread that make seek an alternative framework.

what i have used and really liked for historical backtesting is pytables, it's pretty awesome--can natively treat a portions (amount currently in RAM) of a database as a numpy array and has some real speedy and easy to use c based find/matching functions. i've used TA-Lib wrapped in python (but actually run in c) to populate the tables in the case that i wanted some indicators

plus, its python ; so we talking rapid development times.

matplotlib and scipy have some cool charting capabilities including candlesticks in the finance module.

not sure if it can be used with ms .NET's iron python distribution, i've only used it with linux

PyTables - Getting the most *out* of your data
An Introduction to PyTables video tutorial - PyTables Tutorials - Learn: programming_tools, interface, extremely_large_datasets, pytables, HDF5, efficient, tutorial, tools, cover, data, solution, interfaces, free, videos, video, fast, features, infor

Numpy Home Page
matplotlib: python plotting — Matplotlib v0.98.6svn documentation
TA-Lib : Technical Analysis Library - Home

[marcj]

Generally, you sign legal agreements when you buy data, not to destribute it to others. So is this legal?