How to use the depccg.utils.normalize function in depccg

To help you get started, we’ve selected a few depccg examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github masashi-y / depccg / depccg / tools / data.py View on Github external
def _traverse(self, tree):
        if tree.is_leaf:
            self.cats[str(tree.cat)] += 1
            word = utils.normalize(tree.word)
            self.words[word.lower()] += 1
            for f in get_suffix(word):
                self.suffixes[f] += 1
            for f in get_prefix(word):
                self.prefixes[f] += 1
        else:
            children = tree.children
            if len(children) == 1:
                rule = str(tree.cat), str(children[0].cat)
                self.unary_rules[rule] += 1
                self._traverse(children[0])
            else:
                rule = str(children[0].cat), str(children[1].cat)
                self.seen_rules[rule] += 1
                self._traverse(children[0])
                self._traverse(children[1])