This paper reports a visual tracking system that can track moving objects in real-time with a modest workstation equipped with a pan-tilt device. The algorithm essentially has three parts: (1) feature detection, (2) tracking and (3) control of the robot head. Corners are viewpoint invariant, hence being utilised as the beacon for tracking. Tracking is performed in two stages of Kalman filtering and affine transformation. A technique of reducing greatly the computational time for the correlaton is also described. The Kalman filter predicts intelligently the fovea window and reduced computation dramatically. The affine transformation deals with the unexpected events when there is partial occlusion.