We express the Cartan-Munzner polynomials for isoparametric hypersurfaces of Clifford type in terms of the moment map of the spin action. We also give a systematic argument for all the known Cartan-Munzner polynomials of degree four, and express them via the moment map of a certain group action. Since OT-FKM type includes all the non-homogeneous cases, it will give us a clue to investigate the structure of hypersurfaces with higher cohomogeneity group action.