Validatie-rapportage
Vóór elke publicatieronde wordt een representatieve steekproef argumenten per onderwerp handmatig en door een tweede LLM (Gemini, onafhankelijk van het lokale extractiemodel) herbeoordeeld, zie "Transparantie & reproduceerbaarheid" op de Over-pagina. De resultaten (foutpercentage, systematische patronen per stage) komen hier per onderwerp te staan zodra die steekproefronde is afgerond.
Daarnaast draaien we het evalharnas (issue #62) periodiek tegen externe, publiek extern gelabelde referentiedatasets, om extractie- en tagkwaliteit onafhankelijk van onze eigen Kamerdebat-data te meten.
elecdebate60to16: studie vs. AI
Scoringen van dezelfde 318 sprekersbeurten tegen dezelfde gouden data, met hetzelfde model (Qwen/Qwen3.8-27B:ovhcloud) -- per tabel verschilt telkens maar één definitie (argumentherkenning of drogreden-tags). Zie docs/eval-elecdebate.md ("Experiment: definitie- vs. modelbeperking" resp. "Definitieverschil drogredenen") voor de volledige methodologie.
Argumentherkenning: vindt de AI dezelfde tekstspannen als de studie?
| Vergelijking | Precision | Recall | F1 |
|---|---|---|---|
| Studie vs. AI (onze argumentdefinitie) | 25% | 66% | 37% |
| Studie vs. AI + argumentdefinitie van de studie zelf | 26% | 83% | 40% |
Alleen door de brondataset-definitie gevonden (elecdebate60to16-guideline) -- Onze eigen definitie mist deze (fout-negatief), de brondataset-definitie vindt ze wel.
Nine million people—nine million people lost their jobs. Five million people lost their homes. And $13 trillion in family wealth was wiped out. Now, we have come back from that abyss
Five million people lost their homes. And $13 trillion in family wealth was wiped out. Now, we have come back from that abyss
$13 trillion in family wealth was wiped out. Now, we have come back from that abyss
we need to have strong growth, fair growth, sustained growth. We also have to look at how we help families balance the responsibilities at home and the responsibilities at business. So we have a very robust set of plans
We also have to look at how we help families balance the responsibilities at home and the responsibilities at business. So we have a very robust set of plans
Door beide definities gemist (elecdebate60to16 en elecdebate60to16-guideline) -- Ook de lossere brondataset-definitie vindt deze niet -- wijst op een echte modelbeperking, geen definitieverschil.
it has not been easy. So we’re now on the precipice of having a potentially much better economy, but the last thing we need to do is to go back to the policies that failed us in the first place
You called it the gold standard of trade deals
you live in your own reality, but that is not the facts
that is not the facts. The facts are—I did say I hoped it would be a good deal
I built an unbelievable company. Some of the greatest assets anywhere in the world, real estate assets anywhere in the world, beyond the United States, in Europe, lots of different places
Extra fout-positief bij de brondataset-definitie (elecdebate60to16-guideline) -- Door de brondataset-definitie ten onrechte als argument herkend.
Independent experts have looked at what I’ve proposed and looked at what Donald’s proposed, and basically they’ve said this, that if his tax plan, which would blow up the debt by over $5 trillion and would in some instances disadvantage middle-class families compared to the wealthy, were to go into effect, we would lose 3.5 million jobs and maybe have another recession.
Donald thinks that climate change is a hoax perpetrated by the Chinese. I think it’s real.
Incomes went up for everybody. Manufacturing jobs went up also in the 1990s, if we’re actually going to look at the facts.
I’m going to have a special prosecutor. We’re going to enforce the trade deals we have, and we’re going to hold people accountable.
We also, though, need to have a tax system that rewards work and not just financial transactions.
Drogreden-tags: kent de AI dezelfde drogreden toe als de studie?
Losstaande vraag van argumentherkenning hierboven (vandaar een eigen tabel): dit classificeert de gouden drogreden-citaten van de studie zelf, met steeds dezelfde (strict) extractie -- alleen de tag-omschrijving verschilt.
| Vergelijking | Precision | Recall | F1 |
|---|---|---|---|
| Studie vs. AI (onze tag-omschrijving) | 74% | 50% | 60% |
| Studie vs. AI + tag-omschrijving van de studie zelf | 72% | 64% | 68% |
Alleen door de brondataset-definitie getagd (elecdebate60to16-tag-guideline) -- Onze eigen (engere) tag-omschrijving mist deze (fout-negatief), de brondataset-omschrijving vindt ze wel.
I want us to invest in you. I want us to invest in your future.
trumped-up trickle-down
ripped off
stolen
tremendously
Door beide definities gemist (elecdebate60to16 en elecdebate60to16-tag-guideline) -- Ook de bredere brondataset-omschrijving tagt deze niet -- wijst op een echte modelbeperking, geen definitieverschil.
trumped-up trickle-down
choking
it was actually covered very accurately in the New York Times, which is unusual for the New York Times
racist
beautiful
Extra fout-positief bij de brondataset-definitie (elecdebate60to16-tag-guideline) -- Door de bredere brondataset-omschrijving ten onrechte getagd.
have said that Donald is unfit to be the commander- in-chief.
We have to bring back law and order.
In a place like Chicago, where thousands of people have been killed, thousands over the last number of years, in fact, almost 4,000 have been killed since Barack Obama became president, over—almost 4,000 people in Chicago have been killed.
We’re dying with it
I could call the heads of Wall Street, the heads of every company in America, I would blow away every record.