Since their invention, nanomanipulation systems in scanning electron microscopes (SEMs) have provided researchers with an increasing ability to interact with objects at the nanoscale. However, most nanomanipulators that are capable of generating nanometer displacement operate in an open-loop without suitable feedback mechanisms. In this article, a robust and effective tracking framework for visual servoing applications is presented inside an SEM to achieve more precise tracking manipulation and measurement. A subpixel template matching tracking algorithm based on contour models in the SEM has been developed to improve the tracking accuracy. A feed-forward controller is integrated into the control system to improve the response time. Experimental results demonstrate that a subpixel tracking accuracy is realized. Furthermore, the robustness against clutter can be achieved even in a challenging tracking environment.