df.filter(pl.int_range(pl.len()).is_in([3, 10, 20]))| embedding |
|---|
| array[f64, 3] |
| null |
| [1.0, null, 2.0] |
| [1.0, NaN, 2.0] |
A row is invalid when it is null, or contains a null, NaN or infinite value. UMAP can’t place such a row, so polars-umap gives you two options through null_policy.
| embedding |
|---|
| array[f64, 3] |
| null |
| [1.0, null, 2.0] |
| [1.0, NaN, 2.0] |
null_policy="error" (default)Invalid rows raise an error that says how many there are and where the first one is:
--------------------------------------------------------------------------- ComputeError Traceback (most recent call last) Cell In[3], line 1 ----> 1 df.select(pl.col("embedding").umap.fit_transform()) File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/dataframe/frame.py:10674, in DataFrame.select(self, *exprs, **named_exprs) 10670 10671 return ( 10672 self.lazy() 10673 .select(*exprs, **named_exprs) > 10674 ._collect_eager(optimizations=QueryOptFlags._eager()) 10675 ) File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/lazyframe/frame.py:1949, in LazyFrame._collect_eager(self, optimizations) 1945 def _collect_eager( 1946 self, *, optimizations: QueryOptFlags = DEFAULT_QUERY_OPT_FLAGS 1947 ) -> DataFrame: 1948 """Collect an internal eager operation locally.""" -> 1949 return _eager_engine().collect( 1950 self, optimizations=optimizations, background=False 1951 ) File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/lazyframe/engine.py:410, in _LocalEngine.collect(self, lf, optimizations, background, post_opt_callback) 408 if post_opt_callback is not None: 409 callback = post_opt_callback --> 410 return wrap_df(ldf.collect(self.name, callback)) ComputeError: the plugin failed with message: 3 of 50 rows are null or contain null, NaN or infinite values (first at row 3); clean the data first or pass null_policy="ignore" to skip these rows This error occurred in the following expression: col("embedding")./Users/jeroen/repos/mine/polars-umap/python/polars_umap/_internal.abi3.so:umap_fit_transform()
null_policy="ignore"UMAP is fitted on the valid rows only, and the invalid rows get a null result. The output still has one row per input row, so it lines up with the rest of the frame:
out = df.with_columns(
xy=pl.col("embedding").umap.fit_transform(null_policy="ignore", random_state=0)
)
out.filter(pl.col("xy").is_null())| embedding | xy |
|---|---|
| array[f64, 3] | array[f32, 2] |
| null | null |
| [1.0, null, 2.0] | null |
| [1.0, NaN, 2.0] | null |
The valid rows get exactly the embedding they would get if the invalid rows weren’t there.
knn() supports the same policy. Ignored rows get a null result, and they never appear as someone else’s neighbor:
| embedding |
|---|
| struct[2] |
| {[0, 37, 1],[0.0, 0.442229, 0.490961]} |
| {[1, 37, 0],[0.0, 0.486851, 0.490961]} |
| {[2, 40, 8],[0.0, 0.940555, 0.943256]} |
| null |
| {[4, 36, 41],[0.0, 0.901711, 1.020212]} |
An all-zero vector has no direction, so its cosine distance to anything is undefined. Text embeddings of empty documents are a common source of them. Turn them into nulls first: