Skip to main content

One UMAP per group

Fit a separate UMAP for every group with .over() or group_by().agg().

UMAP is fitted on all rows that an expression sees. In a window or group context, an expression sees one group at a time, so you get an independent UMAP per group, all within a single query.

Say you have embeddings for 20 customers (“tenants”) that should never be mixed:

df.group_by("tenant").len().sort("tenant").head(3)
shape: (3, 2)
tenantlen
i32u32
0500
1500
2500

With .over()

.over() keeps every row in place and adds the per-group embedding next to it:

import polars_umap  # noqa: F401

out = df.with_columns(
    xy=pl.col("embedding").umap.fit_transform(random_state=0).over("tenant"),
)
out.head(3)
shape: (3, 3)
tenantembeddingxy
i32array[f32, 16]array[f32, 2]
0[-10.431762, 8.462978, … 5.161385][6.005804, -2.867212]
0[-0.744056, -0.801295, … -6.763013][-4.95674, 2.645201]
0[0.649313, 2.792154, … -7.325693][-4.557876, 0.355956]

Each group gets exactly the result it would get on its own:

one = df.filter(pl.col("tenant") == 7).select(
    pl.col("embedding").umap.fit_transform(random_state=0)
)
out.filter(pl.col("tenant") == 7)["xy"].equals(one["embedding"], check_names=False)
True

With group_by().agg()

group_by().agg() gives one row per group, with lists of results. explode() turns them back into rows:

per_tenant = df.with_row_index("id").group_by("tenant", maintain_order=True).agg(
    "id", xy=pl.col("embedding").umap.fit_transform(random_state=0)
)
per_tenant.head(3)
shape: (3, 3)
tenantidxy
i32list[u32]list[array[f32, 2]]
0[0, 1, … 499][[6.005804, -2.867212], [-4.95674, 2.645201], … [-3.803444, 2.641677]]
1[500, 501, … 999][[-5.6131, 6.134525], [-4.289757, 4.401392], … [-4.126685, 5.093263]]
2[1000, 1001, … 1499][[-5.831674, -0.289014], [-3.923336, -1.24], … [0.303359, 8.230548]]
per_tenant.explode("id", "xy").head(3)
shape: (3, 3)
tenantidxy
i32u32array[f32, 2]
00[6.005804, -2.867212]
01[-4.95674, 2.645201]
02[-4.557876, 0.355956]

Why this is fast

Polars runs the groups concurrently, and every UMAP shares one thread pool, so the machine is used fully without oversubscribing it. In the benchmarks, one .over() query fitting 1,000 UMAPs took 18 seconds, against 58 seconds for a Python loop over polars-umap.

TipkNN indices are per group too

.umap.knn().over("group") returns neighbor indices relative to the start of each group, not the whole frame. Add a row index per group (pl.int_range(pl.len()).over("group")) to join them back.