How to use the hdfs.util.AsyncWriter function in hdfs

To help you get started, we’ve selected a few hdfs examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github mtth / hdfs / hdfs / client.py View on Github external
buffersize=buffersize,
      )
    loc = res.headers['location']

    def consumer(_data):
      """Thread target."""
      res = self._request(
        method='POST' if append else 'PUT',
        url=loc,
        data=(c.encode(encoding) for c in _data) if encoding else _data,
      )
      if not res:
        raise _to_error(res)

    if data is None:
      return AsyncWriter(consumer)
    else:
      consumer(data)
github mtth / hdfs / hdfs / ext / avro / __init__.py View on Github external
def _start_writer(self):
    _logger.debug('Starting underlying writer.')

    def write(records):
      fastavro.writer(
        fo=self._fo.__enter__(),
        schema=self._schema,
        records=records,
        **self._writer_kwargs
      )

    self._writer = AsyncWriter(write).__enter__()