How to use the tokenizers.BaseTokenizer function in tokenizers

To help you get started, we’ve selected a few tokenizers examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

def _createTokenizer(self):
        return BaseTokenizer(self.source)