A complete Quarto document that embeds text with polars-umap, maps it, and finds similar posts. Download the source and render it yourself.
Author
Jeroen Janssens
Published
November 20, 2026
NoteAbout this example
This page is a regular Quarto document with executable Python cells. Download quarto-report.qmd and render it with:
uv run --with polars-umap --with scikit-learn --with altair --with jupyter \ quarto render quarto-report.qmd
1 Introduction
How do posts from different discussion groups relate to each other? This report embeds posts from three newsgroups as vectors, projects them to 2D with polars-umap, and looks at the structure that emerges. The newsgroups are about hockey, space and religion.
2 Data
Show the code
import altair as altimport numpy as npimport polars as plfrom sklearn.datasets import fetch_20newsgroupsfrom sklearn.decomposition import TruncatedSVDfrom sklearn.feature_extraction.text import TfidfVectorizerimport polars_umap as pupl.Config(fmt_str_lengths=120, tbl_hide_dataframe_shape=True)TOPICS = {"rec.sport.hockey": "Hockey","sci.space": "Space","talk.religion.misc": "Religion",}COLORS = ["#2a78d6", "#eb6834", "#1baf7a"]news = fetch_20newsgroups( subset="all", categories=list(TOPICS), remove=("headers", "footers", "quotes"))
We use the 20 Newsgroups dataset, with headers, signatures and quoted replies removed. Each post becomes a 100-dimensional vector: TF-IDF weights, reduced with a truncated SVD. Any embedding model works the same way from here on.
Show the code
tfidf = TfidfVectorizer(min_df=3, stop_words="english").fit_transform(news.data)vectors = TruncatedSVD(n_components=100, random_state=0).fit_transform(tfidf)posts = pl.DataFrame( {"text": news.data,"topic": [TOPICS[news.target_names[i]] for i in news.target],"embedding": vectors.astype(np.float32), }).with_columns(# Empty posts have an all-zero vector, for which cosine distance is undefined. pl.when(pl.Series(np.linalg.norm(vectors, axis=1) >0)).then(pl.col("embedding")), snippet=pl.col("text").str.replace_all(r"\s+", " ").str.strip_chars().str.slice(0, 120),)
polars-umap projects the vectors to 2D. Cosine distance suits text vectors, whose length mostly reflects the length of the post. null_policy="ignore" skips the empty posts.
Figure 1: Newsgroup posts embedded with polars-umap. Hover over a point to read the start of the post.
Figure 1 shows that hockey and space each form a large region of their own. Religion is more spread out and overlaps with space where posts discuss, for example, cosmology and creation. The small, dense islands at the edges are near-duplicates: repeated announcements and very short posts.
4 How well are the topics separated?
A simple way to quantify the map: for each post, how many of its 10 nearest neighbors in the embedding come from the same newsgroup? polars-umap’s knn() works on the 2D embedding as well.
Table 2: Share of each post’s 10 nearest neighbors on the map that come from the same newsgroup.
topic
same_newsgroup
str
f64
"Hockey"
0.888
"Religion"
0.783
"Space"
0.83
5 Similar posts
Nearest neighbors in the original 100-dimensional space find posts about the same thing. Here are the closest posts to one about the Hubble Space Telescope:
Table 3: The five posts most similar to a post about the Hubble Space Telescope.
distance
topic
snippet
f64
str
str
0.0
"Space"
"Ed Campion Headquarters, Washington, D.C. April 23, 1993 (Phone: 202/358-1780) Kyle Herring Johnson Space Center, Housto"
0.23
"Space"
"Someone from NASA posted that there were very significant mass margins on the HST re-boost mission. A while back i had a"
0.232
"Space"
"It would be nice if someone here from the HST program was talking instead of all the speculation that is going on here. "
0.243
"Space"
"Archive-name: space/acronyms Edition: 8 Acronym List for sci.astro, sci.space, and sci.space.shuttle: Edition 8, 1992 De"
0.245
"Space"
"Has someone actually verified that mass is the predominant constraint on this mission? You seem to be assuming it withou"
0.256
"Space"
"^^^^^^^^^^^^^^^^^^^^^^ I am sure your numbers are far better then mine. As i said above, i don't have exact numbers. How"
The first row is the post itself, at distance 0.
6 Conclusion
A few lines of Polars turn raw text vectors into an interpretable map. The newsgroups about hockey and space form clear clusters. Religion is the most diverse, which matches its “misc” name. The same pipeline works for any embedding column, and with .over() it would produce one map per group just as easily.