How to use the normality.cleaning.is_text function in normality

To help you get started, we’ve selected a few normality examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github pudo / normality / normality / transliteration.py View on Github external
def latinize_text(text, ascii=False):
    """Transliterate the given text to the latin script.

    This attempts to convert a given text to latin script using the
    closest match of characters vis a vis the original script.
    """
    if text is None or not is_text(text) or not len(text):
        return text

    if ascii:
        if not hasattr(latinize_text, '_ascii'):
            latinize_text._ascii = make_transliterator(ASCII_SCRIPT)
        return latinize_text._ascii(text)

    if not hasattr(latinize_text, '_tr'):
        latinize_text._tr = make_transliterator('Any-Latin')
    return latinize_text._tr(text)
github pudo / normality / normality / transliteration.py View on Github external
def ascii_text(text):
    """Transliterate the given text and make sure it ends up as ASCII."""
    text = latinize_text(text, ascii=True)
    if is_text(text):
        return text.encode('ascii', 'ignore').decode('ascii')