Skip to main content

Missing values

How polars-umap treats nulls, NaN and infinite values.

A row is invalid when it is null, or contains a null, NaN or infinite value. UMAP can’t place such a row, so polars-umap gives you two options through null_policy.

df.filter(pl.int_range(pl.len()).is_in([3, 10, 20]))
shape: (3, 1)
embedding
array[f64, 3]
null
[1.0, null, 2.0]
[1.0, NaN, 2.0]

null_policy="error" (default)

Invalid rows raise an error that says how many there are and where the first one is:

df.select(pl.col("embedding").umap.fit_transform())
---------------------------------------------------------------------------
ComputeError                              Traceback (most recent call last)
Cell In[3], line 1
----> 1 df.select(pl.col("embedding").umap.fit_transform())

File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/dataframe/frame.py:10674, in DataFrame.select(self, *exprs, **named_exprs)
  10670 
  10671         return (
  10672             self.lazy()
  10673             .select(*exprs, **named_exprs)
> 10674             ._collect_eager(optimizations=QueryOptFlags._eager())
  10675         )

File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/lazyframe/frame.py:1949, in LazyFrame._collect_eager(self, optimizations)
   1945 def _collect_eager(
   1946     self, *, optimizations: QueryOptFlags = DEFAULT_QUERY_OPT_FLAGS
   1947 ) -> DataFrame:
   1948     """Collect an internal eager operation locally."""
-> 1949     return _eager_engine().collect(
   1950         self, optimizations=optimizations, background=False
   1951     )

File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/lazyframe/engine.py:410, in _LocalEngine.collect(self, lf, optimizations, background, post_opt_callback)
    408 if post_opt_callback is not None:
    409     callback = post_opt_callback
--> 410 return wrap_df(ldf.collect(self.name, callback))

ComputeError: the plugin failed with message: 3 of 50 rows are null or contain null, NaN or infinite values (first at row 3); clean the data first or pass null_policy="ignore" to skip these rows

This error occurred in the following expression:
    col("embedding")./Users/jeroen/repos/mine/polars-umap/python/polars_umap/_internal.abi3.so:umap_fit_transform()

null_policy="ignore"

UMAP is fitted on the valid rows only, and the invalid rows get a null result. The output still has one row per input row, so it lines up with the rest of the frame:

out = df.with_columns(
    xy=pl.col("embedding").umap.fit_transform(null_policy="ignore", random_state=0)
)
out.filter(pl.col("xy").is_null())
shape: (3, 2)
embeddingxy
array[f64, 3]array[f32, 2]
nullnull
[1.0, null, 2.0]null
[1.0, NaN, 2.0]null

The valid rows get exactly the embedding they would get if the invalid rows weren’t there.

Nearest neighbors

knn() supports the same policy. Ignored rows get a null result, and they never appear as someone else’s neighbor:

df.select(pl.col("embedding").umap.knn(k=3, null_policy="ignore")).head(5)
shape: (5, 1)
embedding
struct[2]
{[0, 37, 1],[0.0, 0.442229, 0.490961]}
{[1, 37, 0],[0.0, 0.486851, 0.490961]}
{[2, 40, 8],[0.0, 0.940555, 0.943256]}
null
{[4, 36, 41],[0.0, 0.901711, 1.020212]}
TipZero vectors and cosine distance

An all-zero vector has no direction, so its cosine distance to anything is undefined. Text embeddings of empty documents are a common source of them. Turn them into nulls first:

df.with_columns(
    pl.when(
        pl.col("embedding").arr.to_list().list.eval(pl.element().abs()).list.sum() > 0
    ).then(pl.col("embedding"))
)