convtools

convtools is a Python library that simplifies data transformation by allowing you to define them in a declarative way. It then generates the necessary Python code in the background, saving you time and effort.

Installation

pip install convtools

Documentation

convtools.readthedocs.io

Group by example

from convtools import conversion as c

input_data = [
    {"a": 5, "b": "foo"},
    {"a": 10, "b": "foo"},
    {"a": 10, "b": "bar"},
    {"a": 10, "b": "bar"},
    {"a": 20, "b": "bar"},
]

conv = (
    c.group_by(c.item("b"))
    .aggregate(
        {
            "b": c.item("b"),
            "a_first": c.ReduceFuncs.First(c.item("a")),
            "a_max": c.ReduceFuncs.Max(c.item("a")),
        }
    )
    .pipe(
        c.aggregate({
            "b_values": c.ReduceFuncs.Array(c.item("b")),
            "mode_a_first": c.ReduceFuncs.Mode(c.item("a_first")),
            "median_a_max": c.ReduceFuncs.Median(c.item("a_max")),
        })
    )
    .gen_converter()
)

assert conv(input_data) == {
    'b_values': ['foo', 'bar'],
    'mode_a_first': 10,
    'median_a_max': 15.0
}

Built-in reducers like c.ReduceFuncs.First

* Sum
* SumOrNone
* Max
* MaxRow
* Min
* MinRow
* Count
* CountDistinct
* First
* Last
* Average
* Median
* Percentile
* Mode
* TopK
* Array
* ArrayDistinct
* ArraySorted

DICT REDUCERS ARE IN FACT AGGREGATIONS THEMSELVES, BECAUSE VALUES GET REDUCED.
* Dict
* DictArray
* DictSum
* DictSumOrNone
* DictMax
* DictMin
* DictCount
* DictCountDistinct
* DictFirst
* DictLast

AND LASTLY YOU CAN DEFINE YOUR OWN REDUCER BY PASSING ANY REDUCE FUNCTION
OF TWO ARGUMENTS TO ``c.reduce``.

What's the point if there are tools like Pandas / Polars?

• convtools doesn't need to wrap data in a container to provide functionality, it simply runs the python code it generates on any input
• convtools is lightweight (though optional black is highly recommended for pretty-printing generated code out of curiosity)
• convtools fosters building pipelines on top of iterators, allowing for stream processing
• convtools supports nested aggregations
• convtools is a set of primitives for code generation, so it's just different.

Contributing

The best way to support the development of convtools is to spread the word!

Also, if you already are a convtools user, we would love to hear about your use cases and challenges in the Discussions section.

To report a bug or suggest enhancements, please open an issue and/or submit a pull request.

Reporting a Security Vulnerability: see the security policy.