Template matching and updates are crucial steps in visual object tracking. In this article, we propose a two-stage object tracking algorithm using a dual-template. By design, the initial state of a target can be estimated using a prior fixed template at the first stage with a particle-filter-based tracking framework. The use of prior templates maintains the stability of an object tracking algorithm, because it consists of invariant and important features. In the second step, a mean shift is used to gain the optimal location of the object with the stage update template. The stage template improves the ability of target recognition using a classified update method. The complementary of dual-template improves the quality of template matching and the performance of object tracking. Experimental results demonstrate that the proposed algorithm improves the tracking performance in terms of accuracy and robustness, and it exhibits good results in the presence of deformation, noise and occlusion.