Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

cowcool

Best Strategy Backtesting Software

Recommended Posts

And from the amibroker user group:

-----------------------------------------

Hello All,

 

I have just completed a few days of testing several modeling/backtesting

programs. I thought, perhaps, the other members of the list might find the

results useful; as I am new here, hopefully this can serve as my first

productive contribution.

 

I do a lot of testing & modeling on (a) daily bars (looking to execute intraday

trades) and (b) tick data (for short-term trades). In the case of the first, I

need to create signals on a daily series but execute orders against a 1M

intraday series. So, in order to test an idea over five years of data, I have to

do it over 5Y of 1M data. This gets into performance issues when you start

wanting to run a test on several issues and want to run multiple revisions of

the test. As for testing on tick data, I am sure you are familiar (at least

conceptually) with the performance issues there.

 

I took a look at NinjaTrader, TradeStation, OpenQuant & AmiBroker. I created an

ASCII file of 5Y of EURUSD, USDJPY, GBPUSD, AUDUSD & USDCAD 1M data. In each

program, I ran a simple EMA crossover test (10/50). It was an obnoxious test,

resulting in +300,000 trades, but it was easy to implement and was a good stress

test. What I wanted to do was see if I could get: a) reliable (e.g.

reproducible) results from a single-security test and b) a test of the five FX

pairs as a portfolio (again, in a reliable/reproducible manner).

 

1. Ninja: I use Ninja daily to scalp with and execute some short-term system

code. I had dim hopes for the backtesting since I am familiar with the program -

Ninja really is an execution platform first and an analytical platform second.

The results were more or less what I expected: I could get it to test one issue

with reproducible results at an okay speed (about 3:00M) but it would start to

go into fits when I ran it against all five at once. The results of the five

issue test would vary from instance to instance - it would usually show the

results for the first 3 issues correctly, but on the last 2 it would suffer some

kind of memory issue and give me numbers that were totally off. In one pass, it

even managed to corrupt itself and I had to reload all the data.

 

2. TradeStation: I have a lot of time invested in TradeStation and I was already

familar with it's problems - mainly, that over a large test set, TS will return

different test results. I have talked with TS support and posted on the message

board about this, but I never got anyone interested in what I found to be a

critical issue. The results of this test were as expected: I could not get two

results to match. Any time I would refresh data from the TS data servers and run

the test again, I would get a different result. Sometimes it was as much as test

1 being -$190K and test 2 being +$74K. I do not understand how anyone can use

this tool for successful modeling if they are testing over a large dataset;

just making up a number would have been as useful. I even exported and imported

the data to ensure that it wasn't an issue with the TS data servers. Same

inconsistency. I couldn't test all five pairs together since TS does not do

portfolio backtesting. As

for time of a single test, it is hard to tell with TS as to what, exactly, it

is waiting for/trying to do at any given moment, but I would say it would

usually take about 2/4M per test (although, I have no idea what it was doing in

that 2/4M since the results it returned seemed random at times).

 

3. OpenQuant: This platform looked interesting. I set it up, imported the data

and ran a test. After 20M I noticed that the first EURUSD 1M test was at 10%. I

closed it down and uninstalled it. The performance was just not going to work

for me.

 

4. AmiBroker: You can likely guess the results as I am here as a new member. AB

was able to test EURUSD in about 30 seconds and was able to do the portfolio of

all 5 pairs in about 2M. No matter how many times I ran the test, the numbers it

would return were the same. This fact is pretty crucial to my work. I understand

that data will have errors in it, but I at least need my modeling software to

consistently return results (which AB did). I even created new databases and

populated them with the same data - all test results returned were the same.

 

As you all know, AB was, by far, the cheapest option out of all of the above.

-----------------------------------------------------------

So, nothing to add, just stick to your software and leave the edge to me.

 

Regards

 

Traders37

Share this post


Link to post
Share on other sites
To the O.P....

You don't want to run a live auto strategy on a buggy platform (ninja) or one with bad data (TS).

 

Tradestation has bad data?

 

(I've been very happy with data quality and success of auto-strategies in Tradestation)

 

Thanks.

 

Uli

Edited by Uli Schmuli
Error in post

Share this post


Link to post
Share on other sites
general reading material
800Mb memory perhaps is not enough for NT. you measured computation speed of MC to the windows scratch disk speed really..

 

Tams, could you please tell - you mentioned TT is now supported - can I connect MC to the Guardian (I have XTrader) or do I need connection to FIX server (my broker has no FIX)?

Share this post


Link to post
Share on other sites
800Mb memory perhaps is not enough for NT. you measured computation speed of MC to the windows scratch disk speed really..

 

Tams, could you please tell - you mentioned TT is now supported - can I connect MC to the Guardian (I have XTrader) or do I need connection to FIX server (my broker has no FIX)?

 

 

you better post the question at the TSSupport forum,

they are the authority.

Share this post


Link to post
Share on other sites
Tradestation has bad data?

 

(I've been very happy with data quality and success of auto-strategies in Tradestation)

 

Thanks.

 

Uli

 

Hi Uli:

In backtesting with TradeStation, I have gotten vastly different results before and after deleting my cache. That was running a test on a stock symbol. As the veteran TS user "Goose" says: "The data is always suspect."

 

I don't know if other data sources are more reliable. Is there a thread on "most reliable data providers" for the different types of securities (stocks, futures, forex, options)? Reliable data is the foundation for everything else.

Share this post


Link to post
Share on other sites

I've looked at most packages over the last few years - did an in depth review 2 1/2 years ago for my employer at the time.

 

NeoTicker (the one I got into most) turned out to be a total dog - slow and cumbersome.

AB keeps crashing one me randomly.

Most software can't backtest scans which given that my first trading system was a scanning system I've pretty much given up on other peoples stuff and I'm writing my own stuff in Excel, VBA, VB6, C++ and possibly Smalltalk MT, and no doubt if I get statisically technical I'll use R and MatLab. (I'd probably use Ninja or OpenQuant as Bots if need be). I also use QCollector to download data almost-realtime.

 

At least writing the stuff myself I'm not constrained to other peoples ideas, and it does force me to understand my trading systems.

 

Anyone else using Excel as their main system?

 

If so does any one have a great way of pulling in history into Excel / code?

 

 

--DM

Share this post


Link to post
Share on other sites
I've looked at most packages over the last few years - did an in depth review 2 1/2 years ago for my employer at the time.

 

NeoTicker (the one I got into most) turned out to be a total dog - slow and cumbersome.

AB keeps crashing one me randomly.

Most software can't backtest scans which given that my first trading system was a scanning system I've pretty much given up on other peoples stuff and I'm writing my own stuff in Excel, VBA, VB6, C++ and possibly Smalltalk MT, and no doubt if I get statisically technical I'll use R and MatLab. (I'd probably use Ninja or OpenQuant as Bots if need be). I also use QCollector to download data almost-realtime.

 

At least writing the stuff myself I'm not constrained to other peoples ideas, and it does force me to understand my trading systems.

 

Anyone else using Excel as their main system?

 

If so does any one have a great way of pulling in history into Excel / code?

 

 

--DM

 

 

 

What software you use depends, to a large degree, is dictated by what instrument you trade and the frequencies of your trades...

 

Excel has its place in the game... their charting is actually quite good. It will be a surprise to most people that their charting is very capable and very sophisticated.

 

 

There is a 3rd party add-on that can easily pull EOD data into Excel... the name escapes me at the moment. I will post it if my Alzheimer improves.

Share this post


Link to post
Share on other sites

Dangermouse.

we built a quite intensive excel portfolio backtester. The problem was it takes a long time to run through (45minutes) but the advantage is that you can actually see behind the scenes and also being excel its completely flexible for the user.

The big disadvantage we found was in the data downloading every day became a pain.

Also when building a portfolio testing system you need to ensure the data is all matched up accurately.

We spent a lot of time getting that data as accurate as possible - using futures it was a pain due to the rolling.

What we found now that works far easier is to use the systems that handle data better - eg; esignal, multicharts etc. And run through the signals/alerts . then we get those trades into excel - from there we use excel to really analyse the trades.

Disclaimer - we are not fully automated - while i would like to be, we rather use the indicators to show us possible trades, and we test those for how often they work, what happens when they do and dont work, and then we apply context as a discretionary input.

I am finding Multicharts works pretty well for this - while it does have its issues - its very flexible and it seems the guys there are actively working to be able to link it to excel using more of a cut and paste basis. This would make using their scans great without needing a lot of programming.

Share this post


Link to post
Share on other sites

Hey...

 

When you say we - you work for a fund?

 

"problem...45 minutes" - I do most of my spreadsheet work in VBA or if that's not fast enough VB6 (compiled VBA) or if that's not fast enough C++, for that very reason.

 

Interesting about MultiCharts getting better at Excel linkage. Can it output trades and stops, indicator values at setup time etc for trade system analysis later on?

 

Also can MultiCharts backtest a scanning trading strategy? E.g. trade the best performer from a basket?

 

I'm curious about how people link to Excel. I can use YLoader (yloader.com) to download EOD quotes onto disk, as CSV. I have a COM DLL to load timeseries from disk into arrays in VBA (and into Excel if needed), which if there was demand I'm pondering making a version available as freeware - it does all the memory management so a large timeseries database on disk can be accessed easily without blowing up Excel's 1GB limit.

 

Do you (or does anyone) use intraday bars in Excel?

 

I'm thinking of Sierra charts (due to market profile functionality) - any comments anyone?

 

 

 

-- DM

Share this post


Link to post
Share on other sites
...it does all the memory management so a large timeseries database on disk can be accessed easily without blowing up Excel's 1GB limit.

...

-- DM

 

 

the newer Excel does not have the limit.

Share this post


Link to post
Share on other sites

"you can use EasyLanguage (PRINT) to export/extract anything/anytime to a csv file." - useful to know. I was analysising someone else's trade results and they said they couldn't get out the initial stop or the ATR - so I was restricted in the analysis.

 

the newer Excel does not have the limit - there's still the 2GB limit that 32-bit windows enforces (unless you play with the registry). My intraday database is much bigger than 2GB so having the memory management done for me by the DLL makes it much easier to use...

 

Is there any generic ways to read CSVs - other than the cumbersome import dialog - for let's say I have 100 results and want to summarise them in XL?

 

Cheers

 

-- DM

Share this post


Link to post
Share on other sites
...Is there any generic ways to read CSVs - other than the cumbersome import dialog - for let's say I have 100 results and want to summarise them in XL?

 

Cheers

 

-- DM

 

 

there is a 3rd party add-on that can do it easily... but I don't recall the name right now.

 

 

you can check this code to see if it helps:

 

http://www.traderslaboratory.com/forums/f46/dom-excel-ib-5900.html

10440d1241448157-dom-excel-ib-dom2.gif

89

Share this post


Link to post
Share on other sites

Hi DM -

"we" is myself, and a person i employed to make the coding for excel more efficient.

I know far more about trading than coding.

you asked....

Also can MultiCharts backtest a scanning trading strategy? E.g. trade the best performer from a basket?

Not that i can gather. Thats one of the reasons we used excel. it allowed us to rank instruments for our system/idea testing.

It sounds like you know a fair bit of programming, i would suggest trialling MC - there is a 30 day free trial. There is plenty of info about the language it uses on the web - there are some fantastic people on this and others forum that can help. MC is very flexible and a bit of a pain to learn at the start, but due to this flexibility it seems to be improving all the time.

Given what you have talked about I think you will find it a good system to tie in with your excel. Let it do all the data management, and excel do the sorting.

Share this post


Link to post
Share on other sites

I am on the TS platform and the back testing for the emini is deplorable. You can test at 100% and lose your shirt live. In addition, over night a great strategy will go from 1200.00 profit to 550.00 for two days testing on the weekend with no new data. Go figure.

Just my opinion

Share this post


Link to post
Share on other sites
I am on the TS platform and the back testing for the emini is deplorable. You can test at 100% and lose your shirt live. In addition, over night a great strategy will go from 1200.00 profit to 550.00 for two days testing on the weekend with no new data. Go figure.

Just my opinion

 

 

there is a method to everything...

Share this post


Link to post
Share on other sites
I am on the TS platform and the back testing for the emini is deplorable. You can test at 100% and lose your shirt live. In addition, over night a great strategy will go from 1200.00 profit to 550.00 for two days testing on the weekend with no new data. Go figure.

Just my opinion

 

More detail would be helpful if you are interested in learning how to make TS work for you, instead of just complaining about it. Your comment contains no information useful for troubleshooting your issues.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.