Why Naive $ 1/N $ Diversification Is Not So Naive, and How to Beat It?

We show theoretically that the usual estimated investment strategies will not achieve the optimal Sharpe ratio when the dimensionality is high relative to sample size, and the $ 1/N $ rule is optimal in a 1-factor model with diversifiable risks as dimensionality increases, which explains why it is difficult to beat the $ 1/N $ rule in practice. We also explore conditions under which it can be beaten, and find that we can outperform it by combining it with the estimated rules when $ N $ is small, and by combining it with anomalies or machine learning portfolios, conditional on the profitability of the latter, when $ N $ is large.