It is challenging to remove rain-streaks from a single image because the rain-streaks are spatially varying. Although data-driven rain removal methods have reported promising performance recently, there are still some defects, such as data dependency and insufficient interpretation. A single image deraining algorithm that combines model-based and data-driven approaches is introduced. First, high-frequency information extracted by an improved weighted guided image filter (iWGIF) is used to learn the rain-streaks to avoid interference from other information through the input image. Then, the input and the learned rain streaks are transferred adaptively from the image domain to the feature domain to learn useful features for high quality image deraining. Finally, networks with multi-scale attention mechanisms are used to restore high-quality images from the latent features. Experiments show that the proposed algorithm significantly outperforms state-of-the-art methods in terms of both qualitative and quantitative measures.