Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

cowcool

Best Strategy Backtesting Software

Recommended Posts

And from the amibroker user group:

-----------------------------------------

Hello All,

 

I have just completed a few days of testing several modeling/backtesting

programs. I thought, perhaps, the other members of the list might find the

results useful; as I am new here, hopefully this can serve as my first

productive contribution.

 

I do a lot of testing & modeling on (a) daily bars (looking to execute intraday

trades) and (b) tick data (for short-term trades). In the case of the first, I

need to create signals on a daily series but execute orders against a 1M

intraday series. So, in order to test an idea over five years of data, I have to

do it over 5Y of 1M data. This gets into performance issues when you start

wanting to run a test on several issues and want to run multiple revisions of

the test. As for testing on tick data, I am sure you are familiar (at least

conceptually) with the performance issues there.

 

I took a look at NinjaTrader, TradeStation, OpenQuant & AmiBroker. I created an

ASCII file of 5Y of EURUSD, USDJPY, GBPUSD, AUDUSD & USDCAD 1M data. In each

program, I ran a simple EMA crossover test (10/50). It was an obnoxious test,

resulting in +300,000 trades, but it was easy to implement and was a good stress

test. What I wanted to do was see if I could get: a) reliable (e.g.

reproducible) results from a single-security test and b) a test of the five FX

pairs as a portfolio (again, in a reliable/reproducible manner).

 

1. Ninja: I use Ninja daily to scalp with and execute some short-term system

code. I had dim hopes for the backtesting since I am familiar with the program -

Ninja really is an execution platform first and an analytical platform second.

The results were more or less what I expected: I could get it to test one issue

with reproducible results at an okay speed (about 3:00M) but it would start to

go into fits when I ran it against all five at once. The results of the five

issue test would vary from instance to instance - it would usually show the

results for the first 3 issues correctly, but on the last 2 it would suffer some

kind of memory issue and give me numbers that were totally off. In one pass, it

even managed to corrupt itself and I had to reload all the data.

 

2. TradeStation: I have a lot of time invested in TradeStation and I was already

familar with it's problems - mainly, that over a large test set, TS will return

different test results. I have talked with TS support and posted on the message

board about this, but I never got anyone interested in what I found to be a

critical issue. The results of this test were as expected: I could not get two

results to match. Any time I would refresh data from the TS data servers and run

the test again, I would get a different result. Sometimes it was as much as test

1 being -$190K and test 2 being +$74K. I do not understand how anyone can use

this tool for successful modeling if they are testing over a large dataset;

just making up a number would have been as useful. I even exported and imported

the data to ensure that it wasn't an issue with the TS data servers. Same

inconsistency. I couldn't test all five pairs together since TS does not do

portfolio backtesting. As

for time of a single test, it is hard to tell with TS as to what, exactly, it

is waiting for/trying to do at any given moment, but I would say it would

usually take about 2/4M per test (although, I have no idea what it was doing in

that 2/4M since the results it returned seemed random at times).

 

3. OpenQuant: This platform looked interesting. I set it up, imported the data

and ran a test. After 20M I noticed that the first EURUSD 1M test was at 10%. I

closed it down and uninstalled it. The performance was just not going to work

for me.

 

4. AmiBroker: You can likely guess the results as I am here as a new member. AB

was able to test EURUSD in about 30 seconds and was able to do the portfolio of

all 5 pairs in about 2M. No matter how many times I ran the test, the numbers it

would return were the same. This fact is pretty crucial to my work. I understand

that data will have errors in it, but I at least need my modeling software to

consistently return results (which AB did). I even created new databases and

populated them with the same data - all test results returned were the same.

 

As you all know, AB was, by far, the cheapest option out of all of the above.

-----------------------------------------------------------

So, nothing to add, just stick to your software and leave the edge to me.

 

Regards

 

Traders37

Share this post


Link to post
Share on other sites
To the O.P....

You don't want to run a live auto strategy on a buggy platform (ninja) or one with bad data (TS).

 

Tradestation has bad data?

 

(I've been very happy with data quality and success of auto-strategies in Tradestation)

 

Thanks.

 

Uli

Edited by Uli Schmuli
Error in post

Share this post


Link to post
Share on other sites
general reading material
800Mb memory perhaps is not enough for NT. you measured computation speed of MC to the windows scratch disk speed really..

 

Tams, could you please tell - you mentioned TT is now supported - can I connect MC to the Guardian (I have XTrader) or do I need connection to FIX server (my broker has no FIX)?

Share this post


Link to post
Share on other sites
800Mb memory perhaps is not enough for NT. you measured computation speed of MC to the windows scratch disk speed really..

 

Tams, could you please tell - you mentioned TT is now supported - can I connect MC to the Guardian (I have XTrader) or do I need connection to FIX server (my broker has no FIX)?

 

 

you better post the question at the TSSupport forum,

they are the authority.

Share this post


Link to post
Share on other sites
Tradestation has bad data?

 

(I've been very happy with data quality and success of auto-strategies in Tradestation)

 

Thanks.

 

Uli

 

Hi Uli:

In backtesting with TradeStation, I have gotten vastly different results before and after deleting my cache. That was running a test on a stock symbol. As the veteran TS user "Goose" says: "The data is always suspect."

 

I don't know if other data sources are more reliable. Is there a thread on "most reliable data providers" for the different types of securities (stocks, futures, forex, options)? Reliable data is the foundation for everything else.

Share this post


Link to post
Share on other sites

I've looked at most packages over the last few years - did an in depth review 2 1/2 years ago for my employer at the time.

 

NeoTicker (the one I got into most) turned out to be a total dog - slow and cumbersome.

AB keeps crashing one me randomly.

Most software can't backtest scans which given that my first trading system was a scanning system I've pretty much given up on other peoples stuff and I'm writing my own stuff in Excel, VBA, VB6, C++ and possibly Smalltalk MT, and no doubt if I get statisically technical I'll use R and MatLab. (I'd probably use Ninja or OpenQuant as Bots if need be). I also use QCollector to download data almost-realtime.

 

At least writing the stuff myself I'm not constrained to other peoples ideas, and it does force me to understand my trading systems.

 

Anyone else using Excel as their main system?

 

If so does any one have a great way of pulling in history into Excel / code?

 

 

--DM

Share this post


Link to post
Share on other sites
I've looked at most packages over the last few years - did an in depth review 2 1/2 years ago for my employer at the time.

 

NeoTicker (the one I got into most) turned out to be a total dog - slow and cumbersome.

AB keeps crashing one me randomly.

Most software can't backtest scans which given that my first trading system was a scanning system I've pretty much given up on other peoples stuff and I'm writing my own stuff in Excel, VBA, VB6, C++ and possibly Smalltalk MT, and no doubt if I get statisically technical I'll use R and MatLab. (I'd probably use Ninja or OpenQuant as Bots if need be). I also use QCollector to download data almost-realtime.

 

At least writing the stuff myself I'm not constrained to other peoples ideas, and it does force me to understand my trading systems.

 

Anyone else using Excel as their main system?

 

If so does any one have a great way of pulling in history into Excel / code?

 

 

--DM

 

 

 

What software you use depends, to a large degree, is dictated by what instrument you trade and the frequencies of your trades...

 

Excel has its place in the game... their charting is actually quite good. It will be a surprise to most people that their charting is very capable and very sophisticated.

 

 

There is a 3rd party add-on that can easily pull EOD data into Excel... the name escapes me at the moment. I will post it if my Alzheimer improves.

Share this post


Link to post
Share on other sites

Dangermouse.

we built a quite intensive excel portfolio backtester. The problem was it takes a long time to run through (45minutes) but the advantage is that you can actually see behind the scenes and also being excel its completely flexible for the user.

The big disadvantage we found was in the data downloading every day became a pain.

Also when building a portfolio testing system you need to ensure the data is all matched up accurately.

We spent a lot of time getting that data as accurate as possible - using futures it was a pain due to the rolling.

What we found now that works far easier is to use the systems that handle data better - eg; esignal, multicharts etc. And run through the signals/alerts . then we get those trades into excel - from there we use excel to really analyse the trades.

Disclaimer - we are not fully automated - while i would like to be, we rather use the indicators to show us possible trades, and we test those for how often they work, what happens when they do and dont work, and then we apply context as a discretionary input.

I am finding Multicharts works pretty well for this - while it does have its issues - its very flexible and it seems the guys there are actively working to be able to link it to excel using more of a cut and paste basis. This would make using their scans great without needing a lot of programming.

Share this post


Link to post
Share on other sites

Hey...

 

When you say we - you work for a fund?

 

"problem...45 minutes" - I do most of my spreadsheet work in VBA or if that's not fast enough VB6 (compiled VBA) or if that's not fast enough C++, for that very reason.

 

Interesting about MultiCharts getting better at Excel linkage. Can it output trades and stops, indicator values at setup time etc for trade system analysis later on?

 

Also can MultiCharts backtest a scanning trading strategy? E.g. trade the best performer from a basket?

 

I'm curious about how people link to Excel. I can use YLoader (yloader.com) to download EOD quotes onto disk, as CSV. I have a COM DLL to load timeseries from disk into arrays in VBA (and into Excel if needed), which if there was demand I'm pondering making a version available as freeware - it does all the memory management so a large timeseries database on disk can be accessed easily without blowing up Excel's 1GB limit.

 

Do you (or does anyone) use intraday bars in Excel?

 

I'm thinking of Sierra charts (due to market profile functionality) - any comments anyone?

 

 

 

-- DM

Share this post


Link to post
Share on other sites
...it does all the memory management so a large timeseries database on disk can be accessed easily without blowing up Excel's 1GB limit.

...

-- DM

 

 

the newer Excel does not have the limit.

Share this post


Link to post
Share on other sites

"you can use EasyLanguage (PRINT) to export/extract anything/anytime to a csv file." - useful to know. I was analysising someone else's trade results and they said they couldn't get out the initial stop or the ATR - so I was restricted in the analysis.

 

the newer Excel does not have the limit - there's still the 2GB limit that 32-bit windows enforces (unless you play with the registry). My intraday database is much bigger than 2GB so having the memory management done for me by the DLL makes it much easier to use...

 

Is there any generic ways to read CSVs - other than the cumbersome import dialog - for let's say I have 100 results and want to summarise them in XL?

 

Cheers

 

-- DM

Share this post


Link to post
Share on other sites
...Is there any generic ways to read CSVs - other than the cumbersome import dialog - for let's say I have 100 results and want to summarise them in XL?

 

Cheers

 

-- DM

 

 

there is a 3rd party add-on that can do it easily... but I don't recall the name right now.

 

 

you can check this code to see if it helps:

 

http://www.traderslaboratory.com/forums/f46/dom-excel-ib-5900.html

10440d1241448157-dom-excel-ib-dom2.gif

89

Share this post


Link to post
Share on other sites

Hi DM -

"we" is myself, and a person i employed to make the coding for excel more efficient.

I know far more about trading than coding.

you asked....

Also can MultiCharts backtest a scanning trading strategy? E.g. trade the best performer from a basket?

Not that i can gather. Thats one of the reasons we used excel. it allowed us to rank instruments for our system/idea testing.

It sounds like you know a fair bit of programming, i would suggest trialling MC - there is a 30 day free trial. There is plenty of info about the language it uses on the web - there are some fantastic people on this and others forum that can help. MC is very flexible and a bit of a pain to learn at the start, but due to this flexibility it seems to be improving all the time.

Given what you have talked about I think you will find it a good system to tie in with your excel. Let it do all the data management, and excel do the sorting.

Share this post


Link to post
Share on other sites

I am on the TS platform and the back testing for the emini is deplorable. You can test at 100% and lose your shirt live. In addition, over night a great strategy will go from 1200.00 profit to 550.00 for two days testing on the weekend with no new data. Go figure.

Just my opinion

Share this post


Link to post
Share on other sites
I am on the TS platform and the back testing for the emini is deplorable. You can test at 100% and lose your shirt live. In addition, over night a great strategy will go from 1200.00 profit to 550.00 for two days testing on the weekend with no new data. Go figure.

Just my opinion

 

 

there is a method to everything...

Share this post


Link to post
Share on other sites
I am on the TS platform and the back testing for the emini is deplorable. You can test at 100% and lose your shirt live. In addition, over night a great strategy will go from 1200.00 profit to 550.00 for two days testing on the weekend with no new data. Go figure.

Just my opinion

 

More detail would be helpful if you are interested in learning how to make TS work for you, instead of just complaining about it. Your comment contains no information useful for troubleshooting your issues.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • $TNDM Tandem Diabetes stock top of range breakout watch above 36.19, https://stockconsultant.com/?TNDM
    • $PTN Palatin Technologies stock back to 1.74 support area, https://stockconsultant.com/?PTN
    • $CHWY Chewy stock breakdown watch, https://stockconsultant.com/?CHWY
    • $COF Capital One stock right at the top of the range, breakout watch, https://stockconsultant.com/?COF
    • Date: 25th April 2024. Investors Monitor a Potential Japanese Intervention, and upcoming Tech Earnings. Meta stocks top earnings expectations, but revenue guidance for the next 6 months triggers significant selloff. Meta stocks decline 15.00% and the Magnificent Seven also trade lower. Japanese Authorities are on watch and most market experts predict the Japanese Federal Government will intervene once again. The Japanese Yen is the day’s worst performing currency while the Australian Dollar continues to top the charts. The US Dollar trades 0.10% lower, but this afternoon’s performance is likely to be dependent on the US GDP. USA100 – Meta Stocks Fall 15% On the Next 6-Months Guidance The NASDAQ has declined 1.51% over the past 24 hours, unable to maintain momentum from Monday and Tuesday. Technical analysts advise the decline is partially simply a break in the bullish momentum and the asset continues to follow a bullish correction pattern. However, if the decline continues throughout the day, the retracement scenario becomes a lesser possibility. In terms of indications and technical analysis, most oscillators, and momentum-based signals point to a downward price movement. The USA100 trades below the 75-Bar EMA, below the VWAP and the RSI hovers above 40.00. All these factors point towards a bearish trend. The bearish signals are also likely to strengthen if the price declines below $17,295.11. The stock which is experiencing considerably large volatility is Meta which has fallen more than 15.00%. The past quarter’s earnings beat expectations and according to economists, remain stable and strong. Earnings Per Share beat expectations by 8.10% and revenue was as expected. However, company expenses significantly rose in the past quarter and the guidance for the second half of the year is lower than previous expectations. These two factors have caused investors to consider selling their shares and cashing in their profits. Meta’s decline is one of the main causes for the USA100’s bearish trend. CFRA Senior Analyst, Angelo Zino, advises the selloff may be a slight over reaction based on earnings data. If Meta stocks rise again, investors can start to evaluate a possible upward correction. However, a concern for investors is that more and more companies are indicating caution for the second half of the year. The price movements will largely now depend on Microsoft and Alphabet earnings tonight after market close. Microsoft is the most influential stock for the NASDAQ and Alphabet is the third. The two make up 14.25% of the overall index. If the two companies also witness their stocks decline after the earnings reports, the USA100 may struggle to gain upward momentum. EURJPY – Will Japan Intervene Again? In the currency market, the Japanese Yen remains within the spotlight as investors believe the Japanese Federal Government is likely to again intervene. The Federal Government has previously intervened in the past 12 months which caused a sharp rise in the Yen before again declining. The government opted for this option in an attempt to hinder a further decline. Volatility within the Japanese Yen will also depend on today’s US GDP reading and tomorrow’s Core PCE Price Index. However, investors will more importantly pay close attention to the Bank of Japan’s monetary policy. Investors will be keen to see if the central bank believes it is appropriate to again hike in 2024 as well as comment regarding inflation and the economy. In terms of technical analysis, breakout levels can be considered as areas where the exchange rate may retrace or correct. Breakout levels can be seen at 166.656 and 166.333. However, the only indicators pointing to a decline are the RSI and similar oscillators which advise the price is at risk of being “overbought”. Always trade with strict risk management. Your capital is the single most important aspect of your trading business. Please note that times displayed based on local time zone and are from time of writing this report. Click HERE to access the full HFM Economic calendar. Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding on how markets work. Click HERE to register for FREE! Click HERE to READ more Market news. Michalis Efthymiou Market Analyst HFMarkets Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in FX and CFDs products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.