We explain the rotating polygon instability on a swirling fluid surface [G. H. Vatistas, J. Fluid Mech. 217, 241 (1990)JFLSA70022-1120 and Jansson et al., Phys. Rev. Lett. 96, 174502 (2006)PRLTAO0031-9007] in terms of resonant interactions between gravity waves on the outer part of the surface and centrifugal waves on the inner part. Our model is based on potential flow theory, linearized around a potential vortex flow with a free surface for which we show that unstable resonant states appear. Limiting our attention to the lowest order mode of each type of wave and their interaction, we obtain an analytically soluble model, which, together with estimates of the circulation based on angular momentum balance, reproduces the main features of the experimental phase diagram. The generality of our arguments implies that the instability should not be limited to flows with a rotating bottom (implying singular behavior near the corners), and indeed we show that we can obtain the polygons transiently by violently stirring liquid nitrogen in a hot container.