[2403.19898] Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting