The magnetocaloric effect of a magnetic material is characterized by two quantities, the isothermal entropy change and the adiabatic temperature change, both of which are functions of temperature and applied magnetic field. We discuss the scaling properties of these quantities close to a second-order phase transition within the context of the theory of critical phenomena. Sufficiently close to the critical temperature of a second-order material, the scaling of the isothermal entropy change will be determined by the critical exponents and will be the same as that of the singular part of the entropy itself. However, this is only true in the critical region near Tc and for small fields; for finite fields, scaling with constant exponents, in general, break down, even at Tc. The field dependence can then be described by field-dependent scaling exponents. We show that the scaling exponents at finite fields are not universal, showing significant variation for models in the same universality class. As regards the adiabatic temperature change, it is not determined exclusively by the singular part of the free energy and its derivatives. We show that the field dependence of the adiabatic temperature change at the critical temperature depends on the nonsingular part of the specific heat. The field dependence can still be fitted to a power-law expression but with nonuniversal exponents, as we show explicitly both within mean-field theory and using the so-called Arrott-Noakes equation of state. Within the framework of the Bean-Rodbell model, we briefly consider the scaling properties of the magnetocaloric effect in first-order materials. Finally, we discuss the implications of our findings for a widely used phenomenological scaling procedure for magnetocaloric quantities.