New Open Source Project - MS Sql Server Market Data Colletor.

[Szymon]

HI All,

I have finally got some time here, right now I am installing the latest version of Visual Studio 2008, after that I will start programming a small test application.

I also found an excellent charting component that I will incorporate once the first stage is done to utilize the data.

Dundas Data Visualization Gallery

Simon

[stanlyd]

great idea, I would like to get aboard however I can.
I'm on vacation now tho, be back august 1st.

[Szymon]

Originally Posted by stanlyd great idea, I would like to get aboard however I can. I'm on vacation now tho, be back august 1st.

Hi Stanlyd,

Welcome aboard.

Initially I will need testers to test the application, data, etc...

Simon

[BlowFish]

Originally Posted by Szymon Hi Blowfish, I will take your comments into consideration. The first stage is to build a test application to see if the concept works then I will consider expanding it. Regards Simon

I am a great believer in prototyping Picked that up from my first boss when I needed to write firmware and software to utilise hardware my fellow researchers had designed. He would insist on writing a test suite first and then a prototype. The code would morph into the app anyway. This is one of the reasons I wasn't particularly bowled over by OOP, I was kind of used to writing re usable code.

I do wonder why you picked 1 hour granularity to collect and store? Forgetting tick data for now nearly all applications (and data providers for that matter) provide and store data in chunks of 1 minute. Even if you only want hourly bars it makes sense to collect and store minute data (later perhaps ad tick data) and to aggregate them into hours. The only expense is a little more storage.

I am not familiar with OEC's data feed api but it is likely not to be a question of expanding later but of adding code now to filter and aggregate minute data into hour bars and then removing it later. Hope you don't mind me bleating on about this but I think this is a fundememental design concideration that will actually add complexity to diminish utility.

The visualisation stuff looks nice! A shame that open source stuff is relatively rudimentary though JFreecharts isn't too bad. I know a couple of people who have done great things with it. IB wasn't one of them!

Have fun this weekend with the new software!

[UrmaBlume]

Originally Posted by Szymon Hi All,I'm starting this thread, it is going to be based on open source licence.The aim of this application project is to produce software that will record hourly price of instruments into a MS Sql server. This server can be sitting locally or somewhere else via the web, you can then use .NET xml technologies to retrieve information.The first part of the project is to find out a cheap or free hourly market data provider that has an API feed.What are the advantages of this kind of application? Well for all you system builders out there, You can then develop other applications that can utilize the information and build systems with your formulas and strategies that your third party applications cannot provide.After we will find the right market data source. We will then get into developing the database structure. I will post SQL files here so you can create your own databases. At this stage I am only interested in 1hour timeframe so that will be the focus for this project but you can change that to suit any other time frame.The last part of the project is to develop using vb.net a small application that will utilize the market data API feed and record new bar information into the database. The information that I want to record is OPEN, CLOSE, HIGH and LOW for multiple instruments, with an update feature in case the comptuer has been switched off for a number of hours.I am willing to provide a MS 2008 database that will have web access available but only to those people that will assist me in this project. I will build a .NET web query xml page where you can utilize this information for yourself and your other projects.By having multiple clients talking to the database server, no matter what machine is switched off the database is constantly being updated from all those other machines that are switched on. By helping you have a choice to utilise this information for your own projects.I hope this makes sense. If you have any questions please do not hesitate to ask me. Regards Simon

Why?

Decades of data in every time frame from 1 minute on up is available for every symbol from several different vendors for very little money.

Trade Station at $99 per month, free with just a few trades, makes all of this data available and it can be easily exported from TS into either a csv file or via dll to any database you want.

Plus with all the great open source db's available why MS db?

For Historical data you can get a free trial for TS and download what ever you want.

Is the sole reason for this app to save $99 a month? I don't get it.

[Frank]

usually best to ignore 'hater' posts like this but I will chime in with why..

you are paying Tradestation for their software, not for the data -- which is a commodity and 'should' be free, just like it is for stocks on Yahoo. would you pay $1200 a year to have access to the time of day? of course not, it is a free add-on data feed with your computer.

personally, I used Tradestation for years and moved away from it. I find it more appealing to manipulate data in my own way -- not through Tradestation. I have a solid database of data --- but it would be nice to have a place to reference when you do care to expand your own database of data for new markets or different timeframes. the net net answer to why? because in a world of peer to peer networking through the internet, it SHOULD be accessible for free.

[UrmaBlume]

Originally Posted by Frank usually best to ignore 'hater' posts like this but I will chime in with why.. you are paying Tradestation for their software, not for the data -- which is a commodity and 'should' be free, just like it is for stocks on Yahoo. would you pay $1200 a year to have access to the time of day? of course not, it is a free add-on data feed with your computer. personally, I used Tradestation for years and moved away from it. I find it more appealing to manipulate data in my own way -- not through Tradestation. I have a solid database of data --- but it would be nice to have a place to reference when you do care to expand your own database of data for new markets or different timeframes. the net net answer to why? because in a world of peer to peer networking through the internet, it SHOULD be accessible for free.

Sorry, not a hater post at all.

The db you propose is very limited as to time frames and symbol list. Such a db is only suitable for a very limited trading/research operation. For any "REAL" trading or research venture $1200 per year for unlimited data is very inexpensive and if one's trading or research won't support that, then, indeed, such an ersatz approach would be appropriate.

Of note is that commercial data feeds like Trade Station are at the bottom of the line. The big boys use direct, near zero latency feeds that cost over $20,000 per month. No matter what you build you can't get that for free. The only reason they pay 20k per month is because they make money by using it.

If free one hour data on 10 symbols fulfills your needs then my all means this is the app for you. I run a small private trading and development operation and could neither trade or develop with the data you describe.

[Frank]

yes, historical data fulfills my research needs. Whether you pay nothing, $20k per month or $12 million per year (as my old firm paid in annual investment infrastructure), guess what? its the same data.

you missed the point by trying to pigeon me into one-hour data and condescending about your 'private trading operation' --- whatever bro, I am not talking about data delivery with zero-latency for trading -- I am talking about historical data. there is a difference. there is no edge in zero-latency raw-compute power. my trading platform with 'infintesimally higher than zero latency' is just fine.

in the future, just try to think about the fact that other people might do things differently than you do. there are many different ways to skin a cat and trust me, you don't know all of them. neither do I but the difference is I seem to be aware of this fact.

[UrmaBlume]

Sorry Frank,

My questions regardiing your approach to data collection and managment are about practicality.

I and every professional trader I know would find such an approach to be not practical and that to be able to access unlimited data, either real time or historical, for any symbol or instrument at any time to be well worth what we pay.

Obviously your approach to data collection and management is adequate for your needs so it is the right one for you.

After almost 30 years of professional/management level involvement with the markets, I do, indeed, understand that there are many ways to profit from trading and just didn't understand why anyone would go to the trouble to build such a seemingly inadequate system just to save a few bucks. The answer of course is that it is not inadequate for your needs and thus is perfect for you.

I didn't mean to criticize of belittle your approach, I just didn't understand how anyone could go to that effort to get so little just to save a very few bucks. Now I know and wish you the best of luck.

cheers

[Szymon]

Hi Guys,

Thanks for your valuable input.

At this stage I am happy to take the hourly information, I use this information to calculate future high probability time reversals plus other things and also I wanted to see if this is all possible, to contribute towards this project and share this information like opensource project.

Here is a simple OpenECry connector, you can sign up for an API Login from OpenECry and use that to login to this test system. On the next system l will put some basic functionality on the form, just to see what kind of information they are offering.

Just extract the folder and double click on the OpenECry_Connector.exe file about 26kb. A form will pop up and then put your login details in, then you will get a test message if all went well.

Regards

Simon