We study and compare instability mechanisms of a Mach 5.65 laminar boundary layer tripped by an isolated diamond-shaped trip and by an array of diamond-shaped trips using direct numerical simulations. A low-Reynolds-number experiment, consisting of the trip array (Semper & Bowersox, AIAA J., vol. 55 (3), 2017, pp. 808–817), is used to validate our simulations. Three dynamically prominent flow structures are observed in both trip configurations. These flow structures are the upstream vortex system, the shock system, and the downstream shear layers/counter-rotating streamwise vortices that originate from the top and sides of the trips. Analysis of the power spectral density of pressure reveals the source of instability to be an interaction between the shear layers and the counter-rotating streamwise vortices downstream of both trip configurations. The interaction leads to the formation of hairpin-like structures that eventually break down to turbulent flow. This finding contrasts with that of an isolated cylindrical trip (Subbareddy et al., J. Fluid Mech., vol. 748, 2014, pp. 848–878) where the upstream vortex system is found to be the source of instability. Therefore, the shape of a trip plays an important role in the instability mechanism. Furthermore, dynamic mode decomposition (Rowley et al., J. Fluid Mech., vol. 641, 2009, pp. 115–127; Schmid, J. Fluid Mech., vol. 656, 2010, pp. 5–28) of three-dimensional snapshots of pressure fluctuations unveil globally dominant modes consistent with the power spectral density analysis in both diamond-shaped trip configurations.