Overfitting Risk

Questions

Questions:

  • Can we improve the overall performance by excluding signals during months when assets consistently have performed worse?
  • A valid hypothesis given the variety of pronounced seasonal patterns associated with commodities, e.g. price, volatility, supply, demand etc.

Approach:

  • Identify calendar months when an AiLA asset signal appears to consistently under perform.
  • Block trading signals generated for that asset, if generated during the under performing month.
  • Compare AiLA index performance with and without this bad month selection.

What is Bad?

  • Define a bad month quantitatively, i.e. arbitrary choice.
  • Divide returns by volatility for simple identification and treatment of outliers.
  • Even with 10 years of historical data a given month only corresponds to a limited sample of daily data points.
  • Consistently bad performance during a given month should be of relevant size compared to the statistical error of average monthly return observed.
  • Here we define bad monthly performance as,
    • Month average return/volatility < -0.1
    • Month average return/volatility < 1.5 STD from average return of best 8 other months.
  • Bad month identification applied to assets in an AiLA long/short index, comprising 102 most liquid assets.

Spurious Improvements

  • Rejecting AiLA signals generated during bad months, obtained from data during the same time period, dramatically improve the results.
  • Identifying bad months during period 2011 to 2016, applied to signals during 2017 to 2021, indicates no improvement of the results.
  • Same exercise performed using signals during 2011 to 2016, and bad months identified during the same or later period, yield the same improvement pattern.
  • The case of overfitting by using bad months from the same period as performance evaluation is obvious, however, illustrates large potential impact due to very low signal-to-noise nature of trading strategies.
  • Emphasize importance of rigorous research process due to risk of un-intended and subtitle sources of overfitting!

Conclusions

  • No improvement was observed associated with hypothesis of rejecting signals during months where an AiLA asset signal appears to have consistently performed worse.
  • However, the exercise demonstrates the large impact overfitting can have on trading strategy performance, due to the very low signal-to-noise ratio.
  • The case of overfitting presented here is intended to be obvious, however, the same kind of problem can easily occur un-intentionally in many subtle ways, e.g. from repetitive use of training/validation data in a Machine Learning cycle.
  • The results therefore serves as a reminder of the importance of a rigorous research approach in order to produce reliable results.