In this paper, a novel non circular ROOTMUSIC algorithm for direction of arrival (DOA) estimation in monostatic MIMO radar is proposed. The properties of noncircular signals can be used to form a new virtual array, which enhances the angular resolution. Then a reduce-dimensional transformation matrix is designed to convert the received data into a lower dimensional space, and the DOAs can be obtained by Root-MUSIC algorithm. Compared with RD-ESPRIT method and Root-MUSIC algorithm, the proposed method has better angle estimation performance and handles more targets than RD-ESPRIT. Simulation results are presented to verify the effectiveness of the proposed algorithm.