In theory, the idea of sampling a cohort of shares to use as the Trading Universe for your back-tests, allows you to back-test your system again, once it is finished, against a fresh cohort of shares from the unused part of the sample. The intention is to guard against over-optimising your system based on the first cohort. It also makes back-tests faster. However, in practice, there are difficulties:

1. If you’re only selecting shares from e.g. the LSE share list, you’ll find the number of ‘good’ shares that you’d want to focus your system on is typically only in the 100s, which may not leave many shares after you’ve sampled them
2. As mentioned above, back-testing’s value as a potential predictor of future trading system performance depends to an extent on testing a largish number of shares over an extended period of data (the Law of Large Numbers). Sampling reduces these numbers, which may impact the validity of the back-test
3. Another problem with reducing the number of available shares is that you reduce the number of available trading opportunities, which also increases the risk due to more limited diversification. You can compensate for this to a degree by checking the ‘Adjust trade-size to compensate for list sampling’ option in the List Sampling group of the Test System Dashboard tab, but this is of only limited use. Certainly you should not be surprised if results like CAGR, capital growth etc. are not as good with a sample as it is with the whole population of shares.

A possible alternative approach you may like to consider is as follows. While it has many advantages, the obvious drawback is that you lose the benefits of testing your final system against a fresh set of shares (though you could test it against a different market, e.g. the NYSE if you’ve developed it against e.g. the LSE, but beware of differences in the dynamics of, and share results reported by, these different markets):

1. Use list sampling for rough tests of trading ideas, when the faster back-tests due to the reduced number of shares is a significant advantage
2. Use the full set of shares (i.e. no sampling) for refining your trading spec, once you’ve decided on the basic approach
3. Avoid including anything in your spec that you can’t explain, which you’re only including because you noticed an uptick in the results from a back-test. This is the route to over-optimisation and disappointment, when your Trading Spec underperforms in the future
4. Once your Trading Spec is more or less done, try it against 2 or 3 cohorts of shares by using a sampling ratio of 1:2 or 1:3. You can execute these tests automatically using the Batch Test facility in RuleTrader. Don’t worry too much about the absolute performance; instead focus on the comparative performance between cohorts. If they are wildly different it may indicate over-optimisation. At the least, it would suggest your Spec may not perform well with new data in the future.
5. It’s also a good idea to test your Trading Spec with back-tests over different time periods than the period you used for testing (or a range of sub-periods in that range, if you used a large time range while developing your Spec). Again this can be done automatically using Batch Tests and will highlight any inadvertent temporal over-optimisation.