The masses of the low-lying baryons are evaluated using a total of ten ensembles of dynamical twisted mass fermion gauge configurations. The simulations are performed using two degenerate flavors of light quarks, and a strange and a charm quark fixed to approximately their physical values. The light sea quarks correspond to pseudo scalar masses in the range of about 210 to 430 MeV. We use the Iwasaki improved gluonic action at three values of the coupling constant corresponding to lattice spacing a = 0.094, 0.082 and 0.065 fm determined from the nucleon mass. We check for both finite volume and cutoff effects on the baryon masses. We examine the issue of isospin symmetry breaking for the octet and decuplet baryons and its dependence on the lattice spacing. We show that in the continuum limit isospin breaking is consistent with zero, as expected. We performed a chiral extrapolation of the forty baryon masses using SU(2) chi PT. After taking the continuum limit and extrapolating to the physical pion mass our results are in good agreement with experiment. We provide predictions for the mass of the doubly charmed Xi*(cc), as well as of the doubly and triply charmed Omega s that have not yet been determined experimentally.