The coherence time constitutes one of the most critical parameters that determines whether or not interference is observed in an experiment. For photons, it is traditionally determined by the effective spectral bandwidth of the photon. Here we report on multi-photon interference experiments in which the multi-photon coherence time, defined by the width of the interference signal, depends on the number of interfering photons and on the measurement scheme chosen to detect the particles. A theoretical analysis reveals that all multi-photon interferences with more than two particles feature this dependence, which can be attributed to higher-order effects in the mutual indistinguishability of the particles. As a striking consequence, a single, well-defined many-particle quantum state can exhibit qualitatively different degrees of interference, depending on the chosen observable. Therefore, optimal sensitivity in many-particle quantum interferometry can only be achieved by choosing a suitable detection scheme.