How to use the scrubadub.exceptions.UnicodeRequired function in scrubadub

To help you get started, we’ve selected a few scrubadub examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github datascopeanalytics / scrubadub / tests / test_unicode.py View on Github external
def test_useful_error_message(self):
        try:
            self.clean('John is a byte string')
        except scrubadub.exceptions.UnicodeRequired as e:
            self.assertIn("scrubadub works best with unicode", str(e))
        else:
            self.fail('UnicodeRequired was not raised')
github datascopeanalytics / scrubadub / tests / test_unicode.py View on Github external
def test_not_unicode(self):
        """Make sure unicode works, too"""
        with self.assertRaises(scrubadub.exceptions.UnicodeRequired):
            self.clean('John is a byte string')
github datascopeanalytics / scrubadub / scrubadub / scrubbers.py View on Github external
def clean(self, text, **kwargs):
        """This is the master method that cleans all of the filth out of the
        dirty dirty ``text``. All keyword arguments to this function are passed
        through to the  ``Filth.replace_with`` method to fine-tune how the
        ``Filth`` is cleaned.
        """
        if sys.version_info < (3, 0):  # Only in Python 2, in 3 every string is a Python 2 unicode
            if not isinstance(text, unicode):
                raise exceptions.UnicodeRequired

        clean_chunks = []
        filth = Filth()
        for next_filth in self.iter_filth(text):
            clean_chunks.append(text[filth.end:next_filth.beg])
            clean_chunks.append(next_filth.replace_with(**kwargs))
            filth = next_filth
        clean_chunks.append(text[filth.end:])
        return u''.join(clean_chunks)