Use a pretrained ByteLevelBPETokenizer on text

abdallah197
Fri, 17 Jul 2020 07:52:26 +0000

Hi
I am asking whether there’s a simple way to tokenize a piece of text “I will go to the bedroom” to BPE " I will go to the bed ##room" without training a tokenizer from scratch.

Read full topic