df.group_by("tenant").len().sort("tenant").head(3)| tenant | len |
|---|---|
| i32 | u32 |
| 0 | 500 |
| 1 | 500 |
| 2 | 500 |
UMAP is fitted on all rows that an expression sees. In a window or group context, an expression sees one group at a time, so you get an independent UMAP per group, all within a single query.
Say you have embeddings for 20 customers (“tenants”) that should never be mixed:
.over().over() keeps every row in place and adds the per-group embedding next to it:
import polars_umap # noqa: F401
out = df.with_columns(
xy=pl.col("embedding").umap.fit_transform(random_state=0).over("tenant"),
)
out.head(3)| tenant | embedding | xy |
|---|---|---|
| i32 | array[f32, 16] | array[f32, 2] |
| 0 | [-10.431762, 8.462978, … 5.161385] | [6.005804, -2.867212] |
| 0 | [-0.744056, -0.801295, … -6.763013] | [-4.95674, 2.645201] |
| 0 | [0.649313, 2.792154, … -7.325693] | [-4.557876, 0.355956] |
Each group gets exactly the result it would get on its own:
group_by().agg()group_by().agg() gives one row per group, with lists of results. explode() turns them back into rows:
per_tenant = df.with_row_index("id").group_by("tenant", maintain_order=True).agg(
"id", xy=pl.col("embedding").umap.fit_transform(random_state=0)
)
per_tenant.head(3)| tenant | id | xy |
|---|---|---|
| i32 | list[u32] | list[array[f32, 2]] |
| 0 | [0, 1, … 499] | [[6.005804, -2.867212], [-4.95674, 2.645201], … [-3.803444, 2.641677]] |
| 1 | [500, 501, … 999] | [[-5.6131, 6.134525], [-4.289757, 4.401392], … [-4.126685, 5.093263]] |
| 2 | [1000, 1001, … 1499] | [[-5.831674, -0.289014], [-3.923336, -1.24], … [0.303359, 8.230548]] |
Polars runs the groups concurrently, and every UMAP shares one thread pool, so the machine is used fully without oversubscribing it. In the benchmarks, one .over() query fitting 1,000 UMAPs took 18 seconds, against 58 seconds for a Python loop over polars-umap.
.umap.knn().over("group") returns neighbor indices relative to the start of each group, not the whole frame. Add a row index per group (pl.int_range(pl.len()).over("group")) to join them back.