Use a pretrained ByteLevelBPETokenizer on text

anthony
Fri, 17 Jul 2020 14:20:26 +0000

Hi @abdallah197!

There is a bunch of pre-trained tokenizers in the huggingface/transformers library that you can use directly, without having to train anything. You won’t have any control over how the tokens are split though, as this is based on what the tokenizer learned during training, and the size of its vocabulary. bedroom isn’t really a rare word, so often it will have its own token in the vocabulary.

Your example looks like a WordPiece (instead of BPE), given the ## in ##room which is very specific to this kind of tokenizers. You can try to use those from BERT in the library to see if anything fits your needs, for example with:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

tokenizer.tokenize("I will go to the bedroom")
# ['I', 'will', 'go', 'to', 'the', 'bedroom']

tokenizer.tokenize("I will go to the Bedroom")
# ['I', 'will', 'go', 'to', 'the', 'Bed', '##room']

Read full topic