# Missing values

A row is **invalid** when it is null, or contains a null, NaN or infinite value. UMAP can't place such a row, so polars-umap gives you two options through `null_policy`.


``` python
df.filter(pl.int_range(pl.len()).is_in([3, 10, 20]))
```


shape: (3, 1)

| embedding          |
|--------------------|
| array\[f64, 3\]    |
| null               |
| \[1.0, null, 2.0\] |
| \[1.0, NaN, 2.0\]  |


# `null_policy="error"` (default)

Invalid rows raise an error that says how many there are and where the first one is:


``` python
df.select(pl.col("embedding").umap.fit_transform())
```


    ---------------------------------------------------------------------------
    ComputeError                              Traceback (most recent call last)
    Cell In[3], line 1
    ----> 1 df.select(pl.col("embedding").umap.fit_transform())

    File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/dataframe/frame.py:10674, in DataFrame.select(self, *exprs, **named_exprs)
      10670 
      10671         return (
      10672             self.lazy()
      10673             .select(*exprs, **named_exprs)
    > 10674             ._collect_eager(optimizations=QueryOptFlags._eager())
      10675         )

    File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/lazyframe/frame.py:1949, in LazyFrame._collect_eager(self, optimizations)
       1945 def _collect_eager(
       1946     self, *, optimizations: QueryOptFlags = DEFAULT_QUERY_OPT_FLAGS
       1947 ) -> DataFrame:
       1948     """Collect an internal eager operation locally."""
    -> 1949     return _eager_engine().collect(
       1950         self, optimizations=optimizations, background=False
       1951     )

    File ~/repos/mine/polars-umap/.venv/lib/python3.13/site-packages/polars/lazyframe/engine.py:410, in _LocalEngine.collect(self, lf, optimizations, background, post_opt_callback)
        408 if post_opt_callback is not None:
        409     callback = post_opt_callback
    --> 410 return wrap_df(ldf.collect(self.name, callback))

    ComputeError: the plugin failed with message: 3 of 50 rows are null or contain null, NaN or infinite values (first at row 3); clean the data first or pass null_policy="ignore" to skip these rows

    This error occurred in the following expression:
        col("embedding")./Users/jeroen/repos/mine/polars-umap/python/polars_umap/_internal.abi3.so:umap_fit_transform()


# `null_policy="ignore"`

UMAP is fitted on the valid rows only, and the invalid rows get a null result. The output still has one row per input row, so it lines up with the rest of the frame:


``` python
out = df.with_columns(
    xy=pl.col("embedding").umap.fit_transform(null_policy="ignore", random_state=0)
)
out.filter(pl.col("xy").is_null())
```


shape: (3, 2)

| embedding          | xy              |
|--------------------|-----------------|
| array\[f64, 3\]    | array\[f32, 2\] |
| null               | null            |
| \[1.0, null, 2.0\] | null            |
| \[1.0, NaN, 2.0\]  | null            |


The valid rows get exactly the embedding they would get if the invalid rows weren't there.


# Nearest neighbors

[knn()](../reference/knn.md#polars_umap.knn) supports the same policy. Ignored rows get a null result, and they never appear as someone else's neighbor:


``` python
df.select(pl.col("embedding").umap.knn(k=3, null_policy="ignore")).head(5)
```


shape: (5, 1)

| embedding                                   |
|---------------------------------------------|
| struct\[2\]                                 |
| {\[0, 37, 1\],\[0.0, 0.442229, 0.490961\]}  |
| {\[1, 37, 0\],\[0.0, 0.486851, 0.490961\]}  |
| {\[2, 40, 8\],\[0.0, 0.940555, 0.943256\]}  |
| null                                        |
| {\[4, 36, 41\],\[0.0, 0.901711, 1.020212\]} |


> **Tip: Zero vectors and cosine distance**
>
> An all-zero vector has no direction, so its cosine distance to anything is undefined. Text embeddings of empty documents are a common source of them. Turn them into nulls first:
>
> ``` python
> df.with_columns(
>     pl.when(
>         pl.col("embedding").arr.to_list().list.eval(pl.element().abs()).list.sum() > 0
>     ).then(pl.col("embedding"))
> )
> ```
