A field guide to pulling clean, structured data from every EU institution: 8 platform types, 7 anti-bot walls, and the catalogue of issues we hit (and fixed) across a scan of 8,544 EU URLs.
Brubru runs one config-driven extract engine: give it any EU
institutional listing URL (news, events, publications) and it returns clean,
structured, EuroVoc-tagged items, whichever platform the page is built on. This page
is the practical companion to that engine: how the estate is built, the real-world
problems extraction hits, and how each was fixed. Every item carries the 5-datapoint
contract: public_url, body_txt, body_html,
document_date, creation_date.
Concurrent scan of every URL referenced by Brubru's API source docs.
Two platforms (Drupal + ECL) are ~80% of the estate, so two matching handlers cover most of it. The long tail is anti-bot walls and fully-JS SPAs.
The 429/403 share is partly self-inflicted (28 concurrent workers trip per-host limits) — treat it as an upper bound on "blocked". The hard walls underneath are EUDA (403) and FRA (Anubis proof-of-work).
Of the 1,660 listings, 465 need a browser (top: europarl, FRA, commission); the rest extract from a plain request.
| Platform | Where | Recognise by | Fetch |
|---|---|---|---|
| ECL (Europa Component Library) | *.ec.europa.eu DG sites | .ecl-content-item | requests |
| Drupal / OpenEuropa | agencies (EMA, ECHA, EASA, ACER) | .views-row, .field--name- | requests, some browser |
| WordPress | Joint Undertakings (F4E, rail, H2) | article | requests |
| Power Pages / Dynamics | chips-ju, some JUs | powerappsportal, JS data-grid | browser + render-settle |
| Jalios JCMS | curia (CJEU) | /jcms/ paths | browser |
| SharePoint | ECA | SharePoint markup | browser |
| SPA + search backend | EUIPO, EUR-Lex search | empty shell + XHR | browser |
| Bespoke | long tail | none of the above | content-anchor fallback |
The fetcher escalates: plain request → headless browser → cooldown retry, with TLS-relaxed and render-settle modes. Seven wall types appear across the estate: JS-WAF (EUR-Lex), Anubis proof-of-work (FRA), Cloudflare cooldown, 403/UA blocks (EUDA), cookie handshakes, broken-TLS, JS-faceted search. Genuinely hard walls (EUDA 403, FRA Anubis) remain blocked — Brubru does not try to defeat them.
Symptom → diagnosis → fix. Status: FIXED MITIGATED KNOWN-LIMIT
Light amber = before; blue = after the format-robust parser + detail-page
complete_dates. EIGE's 88% (not 100%) is a few non-article rows; enisa
stays low because its dates are not machine-extractable (the one honest residual).
Symptom: "Europe’s" — apostrophes and accents garbled.
Diagnosis: plain requests defaults an undeclared text/* charset to ISO-8859-1, but EU pages are UTF-8.
Fix: honour the page's real encoding when the charset header is absent. Result: 0 mojibake across the whole estate.
Symptom: "Subscribe to F4E News", "Read more", "Registration" extracted as items and tagged into nonsense.
Diagnosis: the noise filter only ran in the fallback path, not the platform card handlers.
Fix: a centralised junk-title filter at every item-build site. 16/16 junk caught, 0 legit dropped.
Symptom: "Chips JU" → "jute"; "2nd EU-Japan" → "second stage of EMU".
Diagnosis: short titles give the embedder nothing real; absolute cosine score does not separate signal from noise.
Fix: a confidence gate — keep a tag only when several top matches agree on a EuroVoc domain, or the score margin is clear. No tags beats wrong tags. 8/8 garbage cases suppressed, 6/7 real topics kept.
Symptom: "Pacific Islands Forum", "EAEC Treaty" appear as subject tags.
Diagnosis: EuroVoc mixes organisation, institution and treaty names into the same descriptor set as subjects.
Fix: filter by EuroVoc's own domain structure (the organisations + institutions microthesauri + named treaties). 431 names removed; geography ("Community of Madrid") and legal subjects ("judgment of the Court") kept.
Symptom: a Microsoft Power Pages portal (chips-ju Publications) yields nothing.
Diagnosis: the data-grid renders ~12s after load; a short wait snapshots an empty "loading" state.
Fix: a render-settle wait (poll until body text stops growing and no spinner shows) + a one-shot browser retry. chips-ju Publications: 0 → 6, now reliable.
Symptom: even once loaded, the title sits in a <p> and the link is a "Click here" span with no <a>.
Fix: permissive extraction for explicit item-class cards (.publication-item): title = a heading, else the longest paragraph; anchor optional. Scoped to item-class cards so generic chrome is untouched.
Symptom: after I6, JCMS (curia) started returning "About us" / language-switcher items.
Fix: gate permissive extraction to explicit item-class cards only; generic cards keep the strict anchor-required path. Lesson: every relaxation needs a cross-platform regression sweep.
Symptom: event listings (mostly proper-noun names) tag poorly even after I3.
Fix: opt-in ?deep=true fetches each item's detail page and classifies on the full article body. F4E "liquid nitrogen cooling" went from no tags to "reactor cooling system / nuclear fusion".
Symptom: most listings returned items with no date.
Diagnosis (parser): the date regex required "DD Month YYYY" exactly — missing ordinals ("24th June"), US order, and numeric dates.
Diagnosis (location): some Drupal sites carry no date on the listing card at all; it lives only on the article's detail page.
Fix: a format-robust date parser (preferring real date elements) + opt-in ?complete_dates=true to recover the date from the detail page. ACER/agriculture/transport 0% → 100%; eige 0% → 88%; easa 0% → 100%.
Symptom: some source URLs 404 or carry trailing zero-width spaces / http:// (7 zero-width, 31 http, ~48 hard 404s in commission / consilium).
Diagnosis: copy-paste artefacts in the API docs, not engine bugs. The scan flags them for a clean-up pass.
Symptom: dateless "items" that are really form fields (a call-deadline filter, an ecl-datepicker widget).
Fix: strip form, .ecl-datepicker, [role=search], .js-form-item before card selection.
Hard anti-bot walls (EUDA 403, FRA Anubis) stay blocked.
EuroVoc data-modelling quirks: a few bodies (EUISS, "Conference on Disarmament") are filed under topic domains, so they survive the I4 filter on security/defence text.
Non-extractable dates: a small class of fully-JS sites (enisa) hide the date in relative or image form, unrecoverable even when rendered.
Golden URLs per platform (probed N passes; non-determinism is a bug) + an estate-wide concurrent scan that characterises every URL (kind, status, platform, anti-bot) and scores listing extraction (items, date coverage, junk, duplicates, mojibake). Every new bug becomes a regression fixture. Caveat: the 28-worker scan trips some per-host rate limits, so 429/403 counts are an upper bound on "blocked".