Nonlinear optical properties of a regular array of triangular-shaped vortex magnetic nanoparticles is studied using the optical second harmonic generation (SHG) technique. We demonstrate that the SHG azimuthal anisotropy is consistent with the 3m symmetry of individual Co nanodots placed in a square surface lattice. Qualitatively different SHG magnetic hysteresis loops are obtained for circular and linear polarizations of the fundamental radiation. In the first case, a wide SHG hysteresis at zero DC magnetic field H is observed, which is attributed to a macroscopic magnetic toroid moment in Co nanodots induced by a noncentrosymmetric distribution of the magnetization. On the contrary, for the linear pump polarization the SHG loop is similar to observed commonly in linear magnetooptics for vortex magnetic structures and reveals a rather narrow width at H=0. A phenomenological SHG description based on the introduction of the SHG polarization induced by a magnetic toroid moment in vortex magnetic nanostructures is presented.