An Augmented Relative Navigation System (ARNS) is proposed for autonomous satellite formation flying in low-Earth-orbit (LEO). Inter-satellite ranging systems such as those based on radio frequency transmissions can provide additional observation information, e.g. inter-satellite distance measurement, which can be used to increase the Global Positioning System (GPS) stand-alone observation dimension, or treated as a non-linear equality constraint within a smoothly-constrained Kalman filter. Both approaches are implemented in the proposed ARNS described in this paper. An innovative phase integer ambiguity fixing and feedback scheme is implemented to increase the ambiguity fix rate of the GPS carrier phase measurements. A set of Gravity Recovery and Climate Experiment (GRACE) flight data is used to test and validate the relative navigation performance of the proposed methods. Results indicate that the augmented system can improve relative positioning accuracy by an order of magnitude.