Open-air optical flow estimation is crucial for autonomous driving and traffic monitoring. However, it is still challenging to obtain high-precision optical flow under severe imaging environments. To alleviate this problem, we propose a self-adaptive optical flow estimation framework for harsh outdoor scenes, which consists of a frame-adaptive learnable module (FAL-module), a streak-invariant feature extractor (SFE), and a feature matching and flow propagation network. In the framework, three components are jointly learned in an end-to-end manner. The input frames can be adaptively enhanced without sacrificing optical flow-related information by FAL-module. And the SFE decreases the influence of streak-like noise generated by rainy and snowy. Compared with the state-of-the-art methods, the experimental results show that our approach can significantly improve the optical flow estimation quality in multiple harsh outdoor scenes without performance penalties in normal scenes.