How to use the lineflow.apply function in lineflow

To help you get started, we’ve selected a few lineflow examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github tofunlp / lineflow / examples / small_parallel_enja_pytorch.py View on Github external
@lf.apply('ja')
def tokenize(x):
    return [START_TOKEN] + x.split() + [END_TOKEN]
github tofunlp / lineflow / examples / small_parallel_enja_pytorch.py View on Github external
@lf.apply('en')
@lf.apply('ja')
def tokenize(x):
    return [START_TOKEN] + x.split() + [END_TOKEN]
github tofunlp / lineflow / examples / small_parallel_enja_pytorch.py View on Github external
def get_indexer(key, token_to_index, unk_index):
    def indexer(token_to_index, unk_index, x):
        return [token_to_index.get(token, unk_index) for token in x]
    return lf.apply(key)(partial(indexer, token_to_index, unk_index))