Skip to main content

A Quarto report: mapping newsgroup posts

A complete Quarto document that embeds text with polars-umap, maps it, and finds similar posts. Download the source and render it yourself.
Author

Jeroen Janssens

Published

November 20, 2026

NoteAbout this example

This page is a regular Quarto document with executable Python cells. Download quarto-report.qmd and render it with:

uv run --with polars-umap --with scikit-learn --with altair --with jupyter \
  quarto render quarto-report.qmd

1 Introduction

How do posts from different discussion groups relate to each other? This report embeds posts from three newsgroups as vectors, projects them to 2D with polars-umap, and looks at the structure that emerges. The newsgroups are about hockey, space and religion.

2 Data

Show the code
import altair as alt
import numpy as np
import polars as pl
from sklearn.datasets import fetch_20newsgroups
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

import polars_umap as pu

pl.Config(fmt_str_lengths=120, tbl_hide_dataframe_shape=True)

TOPICS = {
    "rec.sport.hockey": "Hockey",
    "sci.space": "Space",
    "talk.religion.misc": "Religion",
}
COLORS = ["#2a78d6", "#eb6834", "#1baf7a"]

news = fetch_20newsgroups(
    subset="all", categories=list(TOPICS), remove=("headers", "footers", "quotes")
)

We use the 20 Newsgroups dataset, with headers, signatures and quoted replies removed. Each post becomes a 100-dimensional vector: TF-IDF weights, reduced with a truncated SVD. Any embedding model works the same way from here on.

Show the code
tfidf = TfidfVectorizer(min_df=3, stop_words="english").fit_transform(news.data)
vectors = TruncatedSVD(n_components=100, random_state=0).fit_transform(tfidf)

posts = pl.DataFrame(
    {
        "text": news.data,
        "topic": [TOPICS[news.target_names[i]] for i in news.target],
        "embedding": vectors.astype(np.float32),
    }
).with_columns(
    # Empty posts have an all-zero vector, for which cosine distance is undefined.
    pl.when(pl.Series(np.linalg.norm(vectors, axis=1) > 0)).then(pl.col("embedding")),
    snippet=pl.col("text").str.replace_all(r"\s+", " ").str.strip_chars().str.slice(0, 120),
)
Show the code
posts.group_by("topic").agg(
    posts=pl.len(), empty=pl.col("embedding").is_null().sum()
).sort("topic")
Table 1: Posts per newsgroup.
topicpostsempty
stru32u32
"Hockey"99927
"Religion"62825
"Space"98733

3 A map of the posts

polars-umap projects the vectors to 2D. Cosine distance suits text vectors, whose length mostly reflects the length of the post. null_policy="ignore" skips the empty posts.

Show the code
posts = posts.umap.fit_transform(
    "embedding", metric="cosine", null_policy="ignore", random_state=0
)
Show the code
(
    alt.Chart(posts.drop_nulls("umap_0").select("topic", "snippet", "umap_0", "umap_1"))
    .mark_circle(size=14, opacity=0.6)
    .encode(
        x=alt.X("umap_0", axis=None),
        y=alt.Y("umap_1", axis=None),
        color=alt.Color(
            "topic:N",
            scale=alt.Scale(domain=list(TOPICS.values()), range=COLORS),
            legend=alt.Legend(title=None, orient="top-left"),
        ),
        tooltip=[alt.Tooltip("topic:N", title="Newsgroup"), alt.Tooltip("snippet:N", title="Post")],
    )
    .properties(width="container", height=480)
    .configure_view(stroke=None)
)
Figure 1: Newsgroup posts embedded with polars-umap. Hover over a point to read the start of the post.

Figure 1 shows that hockey and space each form a large region of their own. Religion is more spread out and overlaps with space where posts discuss, for example, cosmology and creation. The small, dense islands at the edges are near-duplicates: repeated announcements and very short posts.

4 How well are the topics separated?

A simple way to quantify the map: for each post, how many of its 10 nearest neighbors in the embedding come from the same newsgroup? polars-umap’s knn() works on the 2D embedding as well.

Show the code
mapped = posts.drop_nulls("umap_0").with_row_index("row")
neighbors = mapped.select(
    "row", "topic", nn=pu.knn("umap_0", "umap_1", k=11, random_state=0)
).with_columns(nn=pl.col("nn").struct.field("indices").arr.to_list())

(
    neighbors.explode("nn")
    .filter(pl.col("nn") != pl.col("row"))
    .join(mapped.select(nn="row", nn_topic="topic"), on="nn")
    .group_by("topic")
    .agg(same_newsgroup=(pl.col("topic") == pl.col("nn_topic")).mean())
    .sort("topic")
    .with_columns(pl.col("same_newsgroup").round(3))
)
Table 2: Share of each post’s 10 nearest neighbors on the map that come from the same newsgroup.
topicsame_newsgroup
strf64
"Hockey"0.888
"Religion"0.783
"Space"0.83

5 Similar posts

Nearest neighbors in the original 100-dimensional space find posts about the same thing. Here are the closest posts to one about the Hubble Space Telescope:

Show the code
query = (
    posts.with_row_index("row")
    .filter(pl.col("text").str.contains("Hubble") & pl.col("embedding").is_not_null())
    .row(0, named=True)["row"]
)
nn = posts.select(nn=pu.knn("embedding", k=6, metric="cosine", null_policy="ignore", random_state=0))
similar = nn.row(query)[0]
pl.DataFrame({"row": similar["indices"], "distance": similar["distances"]}).join(
    posts.with_row_index("row").select("row", "topic", "snippet"), on="row"
).sort("distance").select("distance", "topic", "snippet").with_columns(pl.col("distance").round(3))
Table 3: The five posts most similar to a post about the Hubble Space Telescope.
distancetopicsnippet
f64strstr
0.0"Space""Ed Campion Headquarters, Washington, D.C. April 23, 1993 (Phone: 202/358-1780) Kyle Herring Johnson Space Center, Housto"
0.23"Space""Someone from NASA posted that there were very significant mass margins on the HST re-boost mission. A while back i had a"
0.232"Space""It would be nice if someone here from the HST program was talking instead of all the speculation that is going on here. "
0.243"Space""Archive-name: space/acronyms Edition: 8 Acronym List for sci.astro, sci.space, and sci.space.shuttle: Edition 8, 1992 De"
0.245"Space""Has someone actually verified that mass is the predominant constraint on this mission? You seem to be assuming it withou"
0.256"Space""^^^^^^^^^^^^^^^^^^^^^^ I am sure your numbers are far better then mine. As i said above, i don't have exact numbers. How"

The first row is the post itself, at distance 0.

6 Conclusion

A few lines of Polars turn raw text vectors into an interpretable map. The newsgroups about hockey and space form clear clusters. Religion is the most diverse, which matches its “misc” name. The same pipeline works for any embedding column, and with .over() it would produce one map per group just as easily.