COSMOS2020: Exploring the Dawn of Quenching for Massive Galaxies at 3 < z < 5 with a New Color-selection Method

We select and characterize a sample of massive (log(M (*)/M (circle dot)) > 10.6) quiescent galaxies (QGs) at 3 < z < 5 in the latest Cosmological Evolution Survey catalog (COSMOS2020). QGs are selected using a new rest-frame color-selection method, based on their probability of belonging to the quiescent group defined by a Gaussian mixture model (GMM) trained on rest-frame colors (NUV - U, U - V, V - J) of similarly massive galaxies at 2 < z < 3. We calculate the quiescent probability threshold above which a galaxy is classified as quiescent using simulated galaxies from the shark semi-analytical model. We find that, at z >= 3 in shark, the GMM/NUVU - VJ method outperforms classical rest-frame UVJ selection and is a viable alternative. We select galaxies as quiescent based on their probability in COSMOS2020 at 3 < z < 5, and compare the selected sample to both UVJ- and NUVrJ-selected samples. We find that, although the new selection matches UVJ and NUVrJ in number, the overlap between color selections is only similar to 50%-80%, implying that rest-frame color commonly used at lower-redshift selections cannot be equivalently used at z > 3. We compute median rest-frame spectral energy distributions for our sample and find the median QG at 3 < z < 5 has a strong Balmer/4000 angstrom break, and residual NUV flux indicating recent quenching. We find the number densities of the entire quiescent population (including post-starbursts) more than doubles from 3.5 +/- 2.2 x 10(-6) Mpc(-3) at 4 < z < 5 to 1.4 +/- 0.4 x 10(-5) Mpc(-3) at 3 < z < 4, confirming that the onset of massive galaxy quenching occurs as early as 3 < z < 5.