huggingface tokenizer本地化

以t5-base为例

本地存储：

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('t5-base')
tokenizer.save_pretrained('your_path')

本地加载：

tokenizer = AutoTokenizer.from_pretrained('your_path')

 

黄世宇/Shiyu Huang's Personal Page：​​https://huangshiyu13.github.io/​​