[2209.06843] Robust field-level inference with dark matter halos