# A Quarto report: mapping newsgroup posts

> **Note: About this example**
>
> This page is a regular Quarto document with executable Python cells. Download [`quarto-report.qmd`](https://github.com/jeroenjanssens/polars-umap/blob/main/gallery/quarto-report.qmd) and render it with:
>
> ``` bash
> uv run --with polars-umap --with scikit-learn --with altair --with jupyter \
>   quarto render quarto-report.qmd
> ```


# 1 Introduction

How do posts from different discussion groups relate to each other? This report embeds posts from three newsgroups as vectors, projects them to 2D with [polars-umap](https://polars-umap.jeroenjanssens.com), and looks at the structure that emerges. The newsgroups are about hockey, space and religion.


# 2 Data


Show the code

``` python
import altair as alt
import numpy as np
import polars as pl
from sklearn.datasets import fetch_20newsgroups
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

import polars_umap as pu

pl.Config(fmt_str_lengths=120, tbl_hide_dataframe_shape=True)

TOPICS = {
    "rec.sport.hockey": "Hockey",
    "sci.space": "Space",
    "talk.religion.misc": "Religion",
}
COLORS = ["#2a78d6", "#eb6834", "#1baf7a"]

news = fetch_20newsgroups(
    subset="all", categories=list(TOPICS), remove=("headers", "footers", "quotes")
)
```


We use the [20 Newsgroups](http://qwone.com/~jason/20Newsgroups/) dataset, with headers, signatures and quoted replies removed. Each post becomes a 100-dimensional vector: TF-IDF weights, reduced with a truncated SVD. Any embedding model works the same way from here on.


Show the code

``` python
tfidf = TfidfVectorizer(min_df=3, stop_words="english").fit_transform(news.data)
vectors = TruncatedSVD(n_components=100, random_state=0).fit_transform(tfidf)

posts = pl.DataFrame(
    {
        "text": news.data,
        "topic": [TOPICS[news.target_names[i]] for i in news.target],
        "embedding": vectors.astype(np.float32),
    }
).with_columns(
    # Empty posts have an all-zero vector, for which cosine distance is undefined.
    pl.when(pl.Series(np.linalg.norm(vectors, axis=1) > 0)).then(pl.col("embedding")),
    snippet=pl.col("text").str.replace_all(r"\s+", " ").str.strip_chars().str.slice(0, 120),
)
```


Show the code

``` python
posts.group_by("topic").agg(
    posts=pl.len(), empty=pl.col("embedding").is_null().sum()
).sort("topic")
```


<figure class="quarto-float quarto-float-tbl figure">


<table class="dataframe table table-sm table-striped small">
<thead>
<tr>
<th>topic</th>
<th>posts</th>
<th>empty</th>
</tr>
<tr>
<th>str</th>
<th>u32</th>
<th>u32</th>
</tr>
</thead>
<tbody>
<tr>
<td>"Hockey"</td>
<td>999</td>
<td>27</td>
</tr>
<tr>
<td>"Religion"</td>
<td>628</td>
<td>25</td>
</tr>
<tr>
<td>"Space"</td>
<td>987</td>
<td>33</td>
</tr>
</tbody>
</table>


<figcaption>Table 1: Posts per newsgroup.</figcaption>
</figure>


# 3 A map of the posts

polars-umap projects the vectors to 2D. Cosine distance suits text vectors, whose length mostly reflects the length of the post. `null_policy="ignore"` skips the empty posts.


Show the code

``` python
posts = posts.umap.fit_transform(
    "embedding", metric="cosine", null_policy="ignore", random_state=0
)
```


Show the code

``` python
(
    alt.Chart(posts.drop_nulls("umap_0").select("topic", "snippet", "umap_0", "umap_1"))
    .mark_circle(size=14, opacity=0.6)
    .encode(
        x=alt.X("umap_0", axis=None),
        y=alt.Y("umap_1", axis=None),
        color=alt.Color(
            "topic:N",
            scale=alt.Scale(domain=list(TOPICS.values()), range=COLORS),
            legend=alt.Legend(title=None, orient="top-left"),
        ),
        tooltip=[alt.Tooltip("topic:N", title="Newsgroup"), alt.Tooltip("snippet:N", title="Post")],
    )
    .properties(width="container", height=480)
    .configure_view(stroke=None)
)
```


<figure class="quarto-float quarto-float-fig figure">

<style>
  #altair-viz-587400e8819b403da08d1d8fdae1cdc7.vega-embed {
    width: 100%;
    display: flex;
  }

  #altair-viz-587400e8819b403da08d1d8fdae1cdc7.vega-embed details,
  #altair-viz-587400e8819b403da08d1d8fdae1cdc7.vega-embed details summary {
    position: relative;
  }
</style>


<figcaption>Figure 1: Newsgroup posts embedded with polars-umap. Hover over a point to read the start of the post.</figcaption>
</figure>


[Figure 1](#fig-map) shows that hockey and space each form a large region of their own. Religion is more spread out and overlaps with space where posts discuss, for example, cosmology and creation. The small, dense islands at the edges are near-duplicates: repeated announcements and very short posts.


# 4 How well are the topics separated?

A simple way to quantify the map: for each post, how many of its 10 nearest neighbors in the embedding come from the same newsgroup? polars-umap's [knn()](../reference/knn.md#polars_umap.knn) works on the 2D embedding as well.


Show the code

``` python
mapped = posts.drop_nulls("umap_0").with_row_index("row")
neighbors = mapped.select(
    "row", "topic", nn=pu.knn("umap_0", "umap_1", k=11, random_state=0)
).with_columns(nn=pl.col("nn").struct.field("indices").arr.to_list())

(
    neighbors.explode("nn")
    .filter(pl.col("nn") != pl.col("row"))
    .join(mapped.select(nn="row", nn_topic="topic"), on="nn")
    .group_by("topic")
    .agg(same_newsgroup=(pl.col("topic") == pl.col("nn_topic")).mean())
    .sort("topic")
    .with_columns(pl.col("same_newsgroup").round(3))
)
```


<figure class="quarto-float quarto-float-tbl figure">


<table class="dataframe table table-sm table-striped small">
<thead>
<tr>
<th>topic</th>
<th>same_newsgroup</th>
</tr>
<tr>
<th>str</th>
<th>f64</th>
</tr>
</thead>
<tbody>
<tr>
<td>"Hockey"</td>
<td>0.888</td>
</tr>
<tr>
<td>"Religion"</td>
<td>0.783</td>
</tr>
<tr>
<td>"Space"</td>
<td>0.83</td>
</tr>
</tbody>
</table>


<figcaption>Table 2: Share of each post's 10 nearest neighbors on the map that come from the same newsgroup.</figcaption>
</figure>


# 5 Similar posts

Nearest neighbors in the original 100-dimensional space find posts about the same thing. Here are the closest posts to one about the Hubble Space Telescope:


Show the code

``` python
query = (
    posts.with_row_index("row")
    .filter(pl.col("text").str.contains("Hubble") & pl.col("embedding").is_not_null())
    .row(0, named=True)["row"]
)
nn = posts.select(nn=pu.knn("embedding", k=6, metric="cosine", null_policy="ignore", random_state=0))
similar = nn.row(query)[0]
pl.DataFrame({"row": similar["indices"], "distance": similar["distances"]}).join(
    posts.with_row_index("row").select("row", "topic", "snippet"), on="row"
).sort("distance").select("distance", "topic", "snippet").with_columns(pl.col("distance").round(3))
```


<figure class="quarto-float quarto-float-tbl figure">


<table class="dataframe table table-sm table-striped small">
<thead>
<tr>
<th>distance</th>
<th>topic</th>
<th>snippet</th>
</tr>
<tr>
<th>f64</th>
<th>str</th>
<th>str</th>
</tr>
</thead>
<tbody>
<tr>
<td>0.0</td>
<td>"Space"</td>
<td>"Ed Campion Headquarters, Washington, D.C. April 23, 1993 (Phone: 202/358-1780) Kyle Herring Johnson Space Center, Housto"</td>
</tr>
<tr>
<td>0.23</td>
<td>"Space"</td>
<td>"Someone from NASA posted that there were very significant mass margins on the HST re-boost mission. A while back i had a"</td>
</tr>
<tr>
<td>0.232</td>
<td>"Space"</td>
<td>"It would be nice if someone here from the HST program was talking instead of all the speculation that is going on here. "</td>
</tr>
<tr>
<td>0.243</td>
<td>"Space"</td>
<td>"Archive-name: space/acronyms Edition: 8 Acronym List for sci.astro, sci.space, and sci.space.shuttle: Edition 8, 1992 De"</td>
</tr>
<tr>
<td>0.245</td>
<td>"Space"</td>
<td>"Has someone actually verified that mass is the predominant constraint on this mission? You seem to be assuming it withou"</td>
</tr>
<tr>
<td>0.256</td>
<td>"Space"</td>
<td>"^^^^^^^^^^^^^^^^^^^^^^ I am sure your numbers are far better then mine. As i said above, i don't have exact numbers. How"</td>
</tr>
</tbody>
</table>


<figcaption>Table 3: The five posts most similar to a post about the Hubble Space Telescope.</figcaption>
</figure>


The first row is the post itself, at distance 0.


# 6 Conclusion

A few lines of Polars turn raw text vectors into an interpretable map. The newsgroups about hockey and space form clear clusters. Religion is the most diverse, which matches its "misc" name. The same pipeline works for any embedding column, and with `.over()` it would produce one map per group just as easily.
