Leitmotif

Solo project for CSE425 (Neural Networks) at BRAC University. There is a live demo you can type into, a paper if you want the whole argument, and all of it on GitHub.

The question

Songs repeat themselves. Choruses come back, riffs loop, and a bridge breaks the pattern on purpose. None of that survives into a caption like “a mellow piano ballad with strings” — the words describe the texture, never the shape.

So: Does a song’s structure tell a model anything the description doesn’t?

I wanted a question where the answer was allowed to be no.

Reading a song as a graph

Each ten-second clip gets cut into one-second segments. Every segment becomes a node holding its log-mel, chroma and MFCC summary. Two kinds of edge join them — one to the segment that follows in time, and one between any pair that sound alike. A chorus that returns at second eight stops being two unrelated moments and becomes a long arc across the graph.

That is the picture above: one real song, its repetitions drawn as arcs.

A graph network reads that shape while BERT reads the caption, and cross-attention lets each side look at the other. Which means you can point at a word and ask which second of audio it was attending to.

Four experiments, in order

#	Question	What it compares
1	Can text alone predict context tags?	BERT vs. guessing at tag frequency
2	Does structure alone predict genre?	GraphSAGE and GAT on segment and chord graphs, vs. a plain CNN
3	Does structure add anything to text?	Graph only, text only, concatenation, cross-attention
4	Can you search music by describing it?	Contrastive dual encoder, InfoNCE

The finding, including the part that failed

On aggregate numbers the audio graph does not beat the text. Masked captions, 50 tags, 2,555 test clips:

Model	Macro-F1
Graph only (audio)	0.227
BERT only (masked caption)	0.468
Early concatenation	0.448
Cross-attention	0.440

Adding the graph made it worse. That is the honest headline, and it is in the paper’s abstract rather than buried in an appendix.

The per-tag picture is where it earns its keep:

Audio helps exactly where a masked caption goes quiet — who is singing, which instruments are playing — and costs accuracy on the tags the surviving words still describe. A single macro-F1 number averages that trade to nothing. The interesting result was never going to be visible in the headline metric.

Search works too, at the scale you’d expect from one laptop GPU: about 20× chance, with the matching clip usually landing in the top 6% of 2,555 candidates. Instead of claiming a listening test I never ran, every retrieved clip is scored by three judges that share no weights with my models — tag overlap, MiniLM caption similarity, and CLAP — each against random clips as a control. All three prefer the retrieved ones.

What fought back

The captions were leaking. 65.6% of tag occurrences appear word-for-word in the caption that was supposed to be predicting them — captions and tags were written by the same musicians in the same sitting. A model can score 0.659 macro-F1 by learning to copy. Every stage-3 number above is from a second training run with those words replaced by [MASK], which costs 0.2 macro-F1 and is the only version worth reporting.

A plain CNN beat my graph. On genre, the CNN reached 0.541 macro-F1 against the best graph model’s 0.485. Letting a small CNN learn each node’s features instead of hand-crafting them closed most of the gap — 0.400 to 0.485 — but not all of it.

Limits

Trained on one RTX 5060 Laptop GPU on ten-second clips at 16 kHz. English captions only, and it inherits MusicCaps’ genre and cultural skew. Retrieval is nowhere near production search. MusicCaps audio comes from YouTube, so 11% of the clips had already disappeared by the time I fetched them.