In this paper, we propose a method to remove the residual speech effects of the channel cepstrum for speaker recognition in the Cepstral Mean Subtraction framework. The proposed Formant-Broadened CMS(FBCMS) is based on the facts that the formants can be found easily in log spectrum which is transformed from the cepstrum and the formants correspond to the dominant poles of all-pole model which is usually modeled vocal tract. The FBCMS evaluates only poles to be broadening from the log spectrum without polynomial factorization and makes a formant-broadened cepstrum by broadening the bandwidths of formant poles. Using 8 simulated telephone channels, we compared the relative errors of estimating channel cepstrum, speaker identification and computational efficiency for CMS, PFCMS, and the proposed method respectively on two databases. The proposed method has shown to yield improved speaker recognition rates with lower computational burden.