Can we improve the overall performance by excluding signals during months when assets consistently have performed worse?
A valid hypothesis given the variety of pronounced seasonal patterns associated with commodities, e.g. price, volatility, supply, demand etc.
Identify calendar months when an AiLA asset signal appears to consistently under perform.
Block trading signals generated for that asset, if generated during the under performing month.
Compare AiLA index performance with and without this bad month selection.
What is Bad?
Define a bad month quantitatively, i.e. arbitrary choice.
Divide returns by volatility for simple identification and treatment of outliers.
Even with 10 years of historical data a given month only corresponds to a limited sample of daily data points.
Consistently bad performance during a given month should be of relevant size compared to the statistical error of average monthly return observed.
Here we define bad monthly performance as,
Month average return/volatility < -0.1
Month average return/volatility < 1.5 STD from average return of best 8 other months.
Bad month identification applied to assets in an AiLA long/short index, comprising 102 most liquid assets.
Rejecting AiLA signals generated during bad months, obtained from data during the same time period, dramatically improve the results.
Identifying bad months during period 2011 to 2016, applied to signals during 2017 to 2021, indicates no improvement of the results.
Same exercise performed using signals during 2011 to 2016, and bad months identified during the same or later period, yield the same improvement pattern.
The case of overfitting by using bad months from the same period as performance evaluation is obvious, however, illustrates large potential impact due to very low signal-to-noise nature of trading strategies.
Emphasize importance of rigorous research process due to risk of un-intended and subtitle sources of overfitting!
No improvement was observed associated with hypothesis of rejecting signals during months where an AiLA asset signal appears to have consistently performed worse.
However, the exercise demonstrates the large impact overfitting can have on trading strategy performance, due to the very low signal-to-noise ratio.
The case of overfitting presented here is intended to be obvious, however, the same kind of problem can easily occur un-intentionally in many subtle ways, e.g. from repetitive use of training/validation data in a Machine Learning cycle.
The results therefore serves as a reminder of the importance of a rigorous research approach in order to produce reliable results.