Abstract: In data assimilation, an ensemble provides a nonintrusive way to evolve a probability density described by a nonlinear prediction model. Although a large ensemble size is required for statis- tical accuracy, the ensemble size is typically limited to a small number due to the computational cost of running the prediction model, which leads to a sampling error. Several methods, such as localization, exist to mitigate the sampling error, often requiring problem-dependent fine-tuning and design. However, many of such methods are applied homogeneously in the physical domain, which is lack of adaptivity. This work introduces another sampling error mitigation method using a smoothness constraint in the Fourier space, and accordingly achieve the inhomogeneous mitigation effect in the physical space. In particular, this work smoothes out the spectrum of the system to increase the stability and accuracy even under a small ensemble size. The efficacy of the new idea is validated through a suite of stringent test problems, including Lorenz 96.