Every time you make a phone call, connect to Wi-Fi, or use GPS, you are relying on a treaty most people have never heard of. The ITU Radio Regulations govern how 193 countries share the radio spectrum — the invisible resource that carries all modern wireless communication. More than 13,000 numbered provisions across three volumes, threaded with cross-references where one rule depends on others scattered hundreds of pages away.
I have spent thirteen years working with this document. Issuing spectrum licences at Sudan’s regulatory authority. Running satellite coordination and interference studies. Sitting in the room at the World Radiocommunication Conference in 2015. These days I co-lead spectrum strategy at stc in Riyadh.
So when I started my MSc at York, I did not go looking for an interesting dataset. I already had a problem I wanted solved: finding the right provision, quickly, without missing the three others it quietly depends on.
Why standard AI tools fall short
The obvious approach is Retrieval-Augmented Generation — RAG — which pulls relevant passages out of a specific document and hands them to a language model, rather than trusting whatever the model happened to absorb during training.
The difficulty is in how the document gets divided. Standard RAG cuts text into fixed-length blocks with no regard for where one rule ends and the next begins. For ordinary prose that is fine. For a treaty it is close to useless. A provision gets split down the middle. A rule is severed from the article that defines its scope. And a cross-reference — the clause that says this applies only subject to No. 9.11 — goes nowhere, because nothing connects the block holding the reference to the block holding the rule it points to. Anyone who has worked with these Regulations knows that following those pointers is the job.
The difficulty is in how the document gets divided. Standard RAG cuts text into fixed-length blocks with no regard for where one rule ends and the next begins. For ordinary prose that is fine. For a treaty it is close to useless.
A structure-aware approach
So my system took the opposite route. Instead of ignoring the document’s structure, it uses it. The ITU publishes the Radio Regulations as Word files, and those files carry hidden formatting markers identifying every article heading, every numbered provision, every footnote. I wrote a parser that reads those markers and rebuilds the document’s real hierarchy.
That made three things possible. Every chunk became one complete provision rather than an arbitrary slice of text. Each provision travelled with its governing article, keeping its legal context intact. And cross-references could be followed automatically — pulling in the connected provisions that a human expert would go and look up anyway.
What the results showed
I built a 150-question benchmark and ran a controlled experiment on York’s Viking supercomputer, comparing five designs and changing one element of the retrieval process at a time.
I had expected an improvement. What I had not expected was how badly the standard approach performs on this kind of text: it put the correct provision first for roughly one question in seventy-five. For anyone doing this work professionally, that is worse than no tool at all, because it looks like an answer. The structure-aware system managed roughly one in five — a sixteenfold improvement. Just as useful was the breakdown. Provision-level chunking alone accounted for most of the gain, which tells anyone building such a system exactly where to start.
I then went beyond the dissertation, testing five language models from 7 billion to 70 billion parameters while holding retrieval constant. Answer quality improved across all five, though the size of the gain varied considerably, with the largest at the 32-billion-parameter model. Stronger models, in other words, do not remove the need for good retrieval. What you feed them still matters.
Stronger models, in other words, do not remove the need for good retrieval. What you feed them still matters.
One result surprised me. A model pretrained specifically on legal text performed worse than a general-purpose model of the same size — which suggests that legal pretraining, built largely on case law and contracts, is no guarantee of competence with the very different language of international treaties.
Personal Reflection
The most valuable lesson of this project had little to do with AI. It came from a parser bug.
Months in, I discovered my code had been silently skipping tables — thousands of rows, including the frequency allocation tables. Anyone in spectrum management will understand the irony: those tables are the part of the Radio Regulations I have consulted most in thirteen years of work, and my system had never seen them. Fixing it meant reparsing the corpus and rerunning every experiment. It was demoralising. It was also the most important thing I did, because everything I had produced until then rested on incomplete data.
That changed how I work. I no longer trust results that arrive too easily, I check the unglamorous foundations before celebrating a headline figure, and I run a five-minute validation before committing five hours of compute.
I also learned the value of negative results. The finding that a legal-domain model underperformed a general one was not what I hoped for. Reporting it honestly rather than quietly dropping it made the work more useful than a tidier story would have been.
Finally, I am grateful to my supervisor, Dr. Majid Latifi, whose feedback consistently pushed me towards greater rigour.
Leave a Reply