New Open Source Project - MS Sql Server Market Data Colletor.

[Szymon]

Hi All,

I'm starting this thread, it is going to be based on open source licence.

The aim of this application project is to produce software that will record hourly price of instruments into a MS Sql server. This server can be sitting locally or somewhere else via the web, you can then use .NET xml technologies to retrieve information.

The first part of the project is to find out a cheap or free hourly market data provider that has an API feed.

What are the advantages of this kind of application? Well for all you system builders out there, You can then develop other applications that can utilize the information and build systems with your formulas and strategies that your third party applications cannot provide.

After we will find the right market data source. We will then get into developing the database structure. I will post SQL files here so you can create your own databases. At this stage I am only interested in 1hour timeframe so that will be the focus for this project but you can change that to suit any other time frame.

The last part of the project is to develop using vb.net a small application that will utilize the market data API feed and record new bar information into the database. The information that I want to record is OPEN, CLOSE, HIGH and LOW for multiple instruments, with an update feature in case the comptuer has been switched off for a number of hours.

I am willing to provide a MS 2008 database that will have web access available but only to those people that will assist me in this project. I will build a .NET web query xml page where you can utilize this information for yourself and your other projects.

By having multiple clients talking to the database server, no matter what machine is switched off the database is constantly being updated from all those other machines that are switched on. By helping you have a choice to utilise this information for your own projects.

I hope this makes sense.

If you have any questions please do not hesitate to ask me.

Regards

Simon

[Szymon]

Hi All,

Had some time today to look into this project.

After finding out the opentick no longer provides their api services to anyone.

I have decided to base this project using openecry API feed as I already have an account with them.

The first step is to find out what their API can do by checking out their documentation and sample software.

After that I will start construction of the database on a Microsoft 2008 SQL Server.

Good trading to you all.

Regards

Simon

[Tams]

how many symbols you will be tracking?

[Szymon]

Here is the API documentation file.

Simon

[Szymon]

Originally Posted by Tams how many symbols you will be tracking?

I think around 10 for now.

But I will put that functionality into the software where you will be able to select how many active instruments to track at any time.

Simon

[BlowFish]

The only thing I would say is try not to limit yourself at the design stage. With data storage cheap and processing power fast, don't make compromises you may later regret. In short even though you are only currently interested in collecting hourly bars for a few instruments, in your design conciderations bear in mind that one day you might require tick data from multiple instruments from multiple exchanges. You are likely to get more support from other contributors too!

[Frank]

I don't know these languages but well-schooled in Excel -- I was wondering what it would take to set-up something on the internet to have some kind of site where anyone who wants it can pull data on futures contracts --- similar to what Yahoo does for stocks/etfs.

Idea is to simply have an efficient site to download data on any contract:
ie,

All 30-min data for

ES Z7
or
NQ H3

just load it by contract once and its done....

thanks for any help/ideas

[Szymon]

Originally Posted by Frank I don't know these languages but well-schooled in Excel -- I was wondering what it would take to set-up something on the internet to have some kind of site where anyone who wants it can pull data on futures contracts --- similar to what Yahoo does for stocks/etfs. Idea is to simply have an efficient site to download data on any contract: ie, All 30-min data for ES Z7 or NQ H3 just load it by contract once and its done.... thanks for any help/ideas

Hi Frank,

Exactly what I am trying to do here. I do not mind sharing a website server and a database to hold that information. And giving access to all users who will contribute to this project.

Regards

Simon

[Szymon]

Originally Posted by BlowFish The only thing I would say is try not to limit yourself at the design stage. With data storage cheap and processing power fast, don't make compromises you may later regret. In short even though you are only currently interested in collecting hourly bars for a few instruments, in your design conciderations bear in mind that one day you might require tick data from multiple instruments from multiple exchanges. You are likely to get more support from other contributors too!

Hi Blowfish,

I will take your comments into consideration. The first stage is to build a test application to see if the concept works then I will consider expanding it.

Regards

Simon

[Szymon]

Project Update,

OK, everybody who is onboard here is an update.

APIFeed for now will be OpenECry.
I have just downloaded the latest version of MS Visual Studio 2009.

Will create a simple application that will login to the OpenECry server and see, how does the data look like and what other fields are there to collect.

I will post this simple app in exe mode for others to use, you will have to have your own API login to test it out.

Now all I have to do is find some time, possibly this weekend.

Regards

Simon