Undergraduate thesis · Informatics, Universitas Ahmad Dahlan
LaguQA
How much do language models actually know about Indonesian songs?
One printed songbook. 107 songs transcribed by hand into ABC 2.1 notation, then turned into 1,200 multiple-choice questions with answer keys. 13 models answered them without ever being trained on this material.
Author Mohammad Farid HendiantoSupervisor Dr. Murinto, M.Kom.Year 2026
No model scored above a guesser that knows no song
This is not one weak model dragging an average down. All 13 models sit below the control line, and the strongest of them falls 2.6 points short.
The benchmark
What is actually being measured
The material is text and musical attributes. No audio file is used as input, so nothing here tests hearing.
Indonesian music is taught in number notation, where the melody is written as the digits 1 to 7 with dots and dashes for octave and duration. Every song in the book was transcribed from that notation into ABC 2.1, a plain-text format a model can read directly. The questions are generated from those transcriptions and from what the book prints on the page, so every answer key traces back to a specific page.
Printed facts
Composer, region of origin, key, time signature, and tempo, taken from what the page states. Where the book states nothing, the correct answer is that it states nothing.
Lyrics
Filling a missing word in a line, and naming the song a line was taken from.
Notation
Naming the song from a fragment of its number notation, with no title given.
Reasoning over notation
Counting bars and finding the highest note in a fragment shown in the question. These are answerable from the fragment alone, so a model that has never seen the song can still get them right.
Results
Thirteen models, and a control that beats all of them
Answers are not read off a letter the model types. For each question the text of all five options is scored by mean log-probability, and the highest option counts as the answer. That removes the advantage a model gets from following the output format.
Multiple choice, 1,200 questions
Accuracy per model. The gold line marks the control: a guesser that has memorised how the answer keys are distributed and knows nothing about any song.
The book is dominated by 4/4 time and the key of Do = C. A guesser that always answers the most common value therefore collects most of the metadata questions for free, which is why 32.1% is the line to read against and not zero. The best untrained model, sahabatai-9b, reaches 29.5%.
Two categories explain most of the gap. On questions that name a song from a fragment of its notation, the whole field stays near the 17.2% random floor. On questions about printed facts, the control alone scores 58.4%, which no untrained model matches.
Does the benchmark respond to the knowledge it claims to measure?
The same base model, before and after LoRA training on the songs. Three seeds of one configuration are shown to give the spread.
Training moves the score from 24.1% to between 52.3% and 61.0%. A benchmark that did not respond this way would be measuring something other than knowledge of the book.
How much of a gap is readable
The three seeds above differ in nothing but the random seed: same training file by sha256, same learning rate, same 2,622 steps. They span 8.7 points, with a standard deviation of 4.4. Any gap smaller than roughly nine points between two single runs cannot be read as a difference in setting. That rule applies to this page as much as to the tables in the thesis.
Where the difficulty sits
Accuracy per question group. The control is shown as its own row because it beats the untrained models on printed facts and loses to them nowhere.
| Question group | Control | sahabatai-9b | gemma4-e2b [lr4e4] seed 1 |
|---|---|---|---|
| Overall | 32.1 | 29.5 | 61.0 |
| Printed facts | 58.4 | 25.3 | 71.2 |
| Title from a clue | 17.8 | 27.1 | 22.4 |
| Lyrics | 24.0 | 25.6 | 40.1 |
| Notation | 17.6 | 17.1 | 46.1 |
| Reasoning | 20.3 | 47.2 | 89.7 |
Data
The questions, and the notation they are keyed against
Every question below is taken from the published file, with its answer key. Pick a category to see one. The questions are written in Indonesian, because that is the language the models are being tested in.
A question from each category
Notation, rendered and played
Staff notation is drawn from the same ABC files that serve as answer keys for the notation questions. What you hear is the material being scored, played through a general soundfont rather than recorded.
Number notation, as printed
Downloads
Everything is public
Limitations
What these numbers do not say
28 of the 107 transcriptions have not passed the beat-conservation and lyric-alignment checks. Notation questions are generated only from the 79 that have.
The time signature of 50 songs is not printed in the book and was inferred from the notation. No time-signature question is built from those songs, because the answer key would then depend on the same reading being tested.
The source is one book. A model that knows Indonesian songs outside it gains nothing here, and a score on LaguQA is not a score on Indonesian music.
The released model was trained on all 107 songs with none held out. It was built to show that the benchmark responds to knowledge, so no number from it may be read as generalisation to unseen songs.
Two different songs in the book are both called Desaku, so a question naming only the title would be ambiguous. A separate column keeps them apart.
Citation
Citing this work
Citing the benchmark means citing the source book as well. Both entries are below.
@misc{laguqa2026,
title = {LaguQA: A Benchmark for Indonesian National and Regional
Song Understanding in Large Language Models},
author = {Hendianto, Mohammad Farid},
year = {2026},
howpublished = {Informatics, Universitas Ahmad Dahlan},
url = {https://laguqa.github.io}
}
@book{pustakabaru2025,
author = {{Tim Pustaka Baru}},
title = {Koleksi Lengkap Lagu-Lagu Daerah dan Wajib Nasional},
publisher = {Pustaka Baru Press},
address = {Banguntapan, Bantul, Yogyakarta},
year = {2025},
isbn = {978-602-0874-22-7}
}