For the problems that the magnetic interference and noise in the magnetometer calibration method result in the heading angle measurement error at present, an improved magnetometer calibration algorithm is presented in this paper. Taking account of the magnetometer installation errors and external magnetic interference, an improved magnetometer error model is established. At the same time, the BP neural network is utilized to train the nonlinear relationship between the magnetic heading and true heading in the soft magnetic interference, thus the nonlinear errors can be reduced. The simulation and experimental verification show that the proposed algorithm can effectively solve the problem of the magnetometer error compensation, reduce the influence of the hard and soft magnetic interference on the heading angle, and more accurate heading angle is obtained. The error is less than ±1°, which certifies the feasibility and effectiveness of the improved calibration algorithm.