The indentation hardness of three different pure forms of silicon was investigated by two different methods. The hardness was probed by direct imaging of the residual impressions and by instrumented indentation using the Oliver–Pharr method. The forms of silicon used were a defective form of amorphous silicon, an amorphous form close to a continuous random network, and a crystalline silicon. The first form deforms via plastic flow and the latter two via phase transition. Two different unloading rates, fast and slow, were used to vary the phase transition behavior. This influenced the relative hardness as measured by instrumented indentation, which is not a reliable method to quantify hardness values in phase transforming materials. Thus, for our phase transforming silicon system, the relative hardness between samples can only be determined correctly by direct imaging, provided that the image accurately reveals the extent of the phase transformed volume.