- Can we improve the overall performance by excluding signals during months when assets consistently have performed worse?
- A valid hypothesis given the variety of pronounced seasonal patterns associated with commodities, e.g. price, volatility, supply, demand etc.
- Identify calendar months when an AiLA asset signal appears to consistently under perform.
- Block trading signals generated for that asset, if generated during the under performing month.
- Compare AiLA index performance with and without this bad month selection.
What is Bad?
- Define a bad month quantitatively, i.e. arbitrary choice.
- Divide returns by volatility for simple identification and treatment of outliers.
- Even with 10 years of historical data a given month only corresponds to a limited sample of daily data points.
- Consistently bad performance during a given month should be of relevant size compared to the statistical error of average monthly return observed.
- Here we define bad monthly performance as,
- Month average return/volatility < -0.1
- Month average return/volatility < 1.5 STD from average return of best 8 other months.
- Bad month identification applied to assets in an AiLA long/short index, comprising 102 most liquid assets.
- Rejecting AiLA signals generated during bad months, obtained from data during the same time period, dramatically improve the results.
- Identifying bad months during period 2011 to 2016, applied to signals during 2017 to 2021, indicates no improvement of the results.
- Same exercise performed using signals during 2011 to 2016, and bad months identified during the same or later period, yield the same improvement pattern.
- The case of overfitting by using bad months from the same period as performance evaluation is obvious, however, illustrates large potential impact due to very low signal-to-noise nature of trading strategies.
- Emphasize importance of rigorous research process due to risk of un-intended and subtitle sources of overfitting!
- No improvement was observed associated with hypothesis of rejecting signals during months where an AiLA asset signal appears to have consistently performed worse.
- However, the exercise demonstrates the large impact overfitting can have on trading strategy performance, due to the very low signal-to-noise ratio.
- The case of overfitting presented here is intended to be obvious, however, the same kind of problem can easily occur un-intentionally in many subtle ways, e.g. from repetitive use of training/validation data in a Machine Learning cycle.
- The results therefore serves as a reminder of the importance of a rigorous research approach in order to produce reliable results.