Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The author disparages ibis but i really think that this is short sighted. Ibis does a great job of mixing sql with dataframes to perform complex queries and abstracts away a lot of the underlyng logic and allows for query optimization.

Example:

df = (

    df.mutate(new_column=df.old_column.dosomething())

      .alias('temp_table')

      .sql('SELECT db_only_function(new_column) AS newer_column  from temp_table')

      .mutate(other_new_column = newer_column.do_other_stuff())

)

It's super flexible and duckdb makes it very performant. The general vice i experience creating overly complex transforms but otherwise it's super useful and really easy to mix dataframes and SQL. Finally it supports pretty much every backend including pyspark and polars



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: