How to use the blingfire.text_to_words function in blingfire

To help you get started, we’ve selected a few blingfire examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github soskek / bookcorpus / tokenize_sentlines.py View on Github external
import sys
from blingfire import text_to_words


for l in sys.stdin:
    if l.strip():
        print(text_to_words(l.strip()))
    else:
        print('')

blingfire

Python wrapper of lightning fast Finite State Machine based NLP library.

MIT
Latest version published 3 years ago

Package Health Score

60 / 100
Full package analysis

Similar packages