# Lazy pipelines

Everything in polars-umap works in lazy mode. The output schema is known up front, so you can build and inspect a query before running it.


``` python
import polars_umap  # noqa: F401

query = (
    pl.scan_parquet(path)
    .filter(pl.col("lang") == "en")
    .umap.fit_transform("embedding", metric="cosine", random_state=0)
    .drop("embedding")
)
query.collect_schema()
```


    Schema([('id', Int64),
            ('lang', String),
            ('umap_0', Float32),
            ('umap_1', Float32)])


``` python
print(query.explain())
```


    simple π 4/4 ["id", "lang", "umap_0", ... 1 other column]
       WITH_COLUMNS:
       [col("__polars_umap_embedding__").arr.get([dyn int: 0]).alias("umap_0"), col("__polars_umap_embedding__").arr.get([dyn int: 1]).alias("umap_1")] 
        simple π 3/3 ["id", "lang", ... 1 other column]
           WITH_COLUMNS:
           [col("embedding")./Users/jeroen/repos/mine/polars-umap/python/polars_umap/_internal.abi3.so:umap_fit_transform().alias("__polars_umap_embedding__")] 
            Parquet SCAN [/var/folders/_l/x6n0w90j54790nrdcb5hfzj40000gp/T/tmpoaypdpsr/embeddings.parquet]
            PROJECT */3 COLUMNS
            SELECTION: col("lang") == "en"
            ESTIMATED ROWS: 5000


The filter runs before UMAP, so only the English rows are embedded. Collect the query, or sink it to a file:


``` python
query.collect().head(3)
```


shape: (3, 4)

| id  | lang | umap_0   | umap_1    |
|-----|------|----------|-----------|
| i64 | str  | f32      | f32       |
| 3   | "en" | 1.251937 | -2.690112 |
| 4   | "en" | -2.06721 | 1.752058  |
| 5   | "en" | 0.657309 | -2.117096 |


``` python
query.sink_parquet("projected.parquet")
```


# Things to keep in mind

- **UMAP needs all rows at once.** It isn't elementwise: the embedding of each row depends on all the others. Polars therefore materializes the input column before calling polars-umap, even in the streaming engine. Filter and select early so that only what's needed is loaded.
- **Filters change the result.** Filtering before UMAP fits a different model than filtering after it. That's usually what you want, but it's worth being deliberate about.
