We present neutron and high-energy synchrotron X-ray diffraction studies to show that the anomalous antiferromagnetic (AF) phase diagram of TmNi2B2C in an applied field along [1 0 0] is governed by a quadrupolar ordering of the Tm ions. The ordering is revealed by a distortion of the lattice with the Tm ions displaced along the c-axis and modulated with the same wave vector QA=(0.484, 0, 0) as the AF phase induced by fields larger than ≈10 kOe. In zero field, the quadrupolar ordering temperature is TQ≅13.5 K but increases to about 20 K in a field of 100 kOe. The Tm displacements are also significantly enhanced, by a factor of 10 at 60 kOe. A model is presented that accounts for the quadrupolar as well as the low-field QF=(0.94, 0.94, 0) and high-field QA=(0.483, 0, 0) AF ordering previously observed in this compound.