How to use the tokenizers.camel.__call__ function in tokenizers

To help you get started, we’ve selected a few tokenizers examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github naturalness / partycrasher / anatomy / lp_files_to_json.py View on Github external
def rec_length(key, value):
        if key not in lengths:
            lengths[key] = []
        if value is None:
            length = 0
        else:
            length = len(camel.__call__(value))
        lengths[key].append((crash['database_id'], length))
        if value is not None:
            #print(key + " " + value + " " + str(length), file=sys.stderr)
            if key not in ulengths:
                ulengths[key] = dict()
            ulengths[key][value] = length