If list sampling is enabled (in the Test System tab of the RuleTrader Test Dashboard) for a back-test, then the sampling process works as follows:

1. The source lists, including the Trading Universe, if used, and any other ShareScope lists that are referenced in your system, are sampled in the sampling ratio defined in the Test Session tab. The exception is ShareScope portfolios, which are never sampled, as it is assumed that, as you have chosen the shares in a portfolio, you are likely to want to test them all. If this is not the case then please make a manually sampled copy of the portfolio and use that instead.
2. The resulting sample of shares are then provided as the source for your Spec’s trading list and benchmark list selectors to operate on
3. If you use a list ranking to pick a specific number or percentage of shares from a list, then the rank size will be adjusted by the sampling ratio to reflect the smaller source list size. This happens automatically, so you do not need to modify the ranking definition.

In theory, the idea of sampling a cohort of shares for your back-tests, allows you to back-test your system, once it is finished, against a fresh cohort of shares from the unused part of the sample. The intention is to guard against over-optimising your system based on the first cohort. It also makes back-tests faster. In practice, there are difficulties:

1. If you’re only selecting shares from e.g. the LSE share list, you’ll find the number of ‘good’ shares that you’d want to focus your system on is typically only in the 100s, which may not leave many shares after you’ve sampled them
2. As mentioned in our article on Back-Testing, we believe its value as a potential predictor of future trading system performance depends, to an extent, on testing a largish number of shares over an extended period of data (the Law of Large Numbers). Sampling reduces these numbers, which may impact the utility of the back-test
3. Another problem with reducing the number of available shares is that you reduce the number of available trading opportunities, which also increases the risk due to more limited diversification. You can compensate for this to a degree by checking the ‘Adjust trade-size to compensate for list sampling’ option in the List Sampling group of the Test System Dashboard tab, but this is of only limited use. Certainly you should not be surprised if results like CAGR, capital growth etc. are not as good with a sample as it is with the whole population of shares.

A possible alternative approach you may like to consider is as follows. While it has many advantages, the obvious drawback is that you lose the benefits of testing your final system against a fresh set of shares (though you could test it against a different market, e.g. the NYSE if you’ve developed it against e.g. the LSE, but beware of differences in the share results reported by different markets):

1. Use list sampling for rough tests of trading ideas, when the faster back-tests due to the reduced number of shares is a significant advantage
2. Use the full set of shares (i.e. no sampling) for refining your trading spec, once you’ve decided on the basic approach
3. Avoid including anything in your spec that you can’t explain, which you’re only including because you noticed an uptick in the results from a back-test. This is the route to over-optimisation and disappointment, when your Trading Spec underperforms in the future
4. Once your Trading Spec is more or less done, try it against 2 or 3 cohorts of shares by using a sampling ratio of 1:2 or 1:3. You can execute these tests automatically using the Batch Test facility in RuleTrader. Don’t worry too much about the absolute performance; instead focus on the comparative performance between cohorts. If they are wildly different it may indicate over-optimisation. At the least, it would suggest your Spec may not perform well with new data in the future.
5. It’s also a good idea to test your Trading Spec with back-tests over different time periods than the period you used for testing (or a range of sub-periods in that range, if you used a large time range while developing your Spec). Again this can be done automatically using Batch Tests and will highlight any inadvertent temporal over-optimisation.