Data is the new constraint in AI. Brubru is built data-first.

The Cellar SPARQL endpoint exposes 7,523 EuroVoc concepts (across 24 languages) and 130+ authority tables. Brubru's nightly job (scripts/sync_eu_authority_labels.py) caches the 12 highest-traffic ones in 6 languages into eu_authority_labels. The chatbot prepends a localised glossary block to its working context whenever an authority URI is in scope, the model sees URI + human label in the user's language at the same time.

• Phase 0-1: authority label cache (18,046 rows across 6 langs) + glossary injector + 4 paginated v1 endpoints (/vocabularies/corporate-bodies + procedures + directories + modification-types).
• Phase 2-3: DCAT-AP self-catalogue (6 seed datasets, 3 serialisations) + OP Core Metadata mixin on every v1 response.
• Phase 4: ELI + CDM ontology pinning + drift CI test (11 static checks).
• Phase 5: IMMC tag column on legislative_carriages (migration 049) backed by Cellar event predicates.
• Phase 6: AKN4EU 4.2 DET renderer for 4 doc types.
• Phase 7-8: EURIO SPARQL client (5 v1 endpoints) + EuroSciVoc resolver.
• Phase 9: OJEEP structural validator for Catalan translations (13 checks per act).
• Phase 11: 51 synthetic queries × 10 categories × 27 distinct CELEX. 51/51 = 100% pass.
• Phase 13: nightly Railway cron + monthly EuroVoc releases diff + freshness gate on /news.

Reference: docs/applications/euvoc.md (canonical living plan). Phase 10 (EuroPass) is deferred until an "EU career" surface lands.

The current generation of the Brubru API (v2) exposes 934 live endpoints (934 endpoints + 6 docs/schema routes). Every data endpoint returns the same five-datapoint contract (public_url, body_txt, body_html, document_date, creation_date) inside one canonical paginated envelope, with the same filters (q full-text, date ranges, order, pagination) on every surface. One key, one schema, one rate limiter, across institutions that otherwise publish in dozens of incompatible ways or publish no machine-readable surface at all. A partner integrates once and reads the whole EU.

Where the data comes from is the whole story. About half the surface is a clean unification of official EU machine-readable APIs; the other half exists only because Brubru built the pipeline, data with no official API anywhere. That second half is the moat. No competitor surfaces it through a single contract. The provenance split is shown just below; the per-family distribution follows it.

API calls are authenticated with a key passed in the X-API-Key header or as Authorization: Bearer <key>. Keys are minted by the admin panel, SHA-256 hashed at rest, and the plaintext is shown once at creation.

• Key format: brubru_live_ followed by 48 hex characters (192 bits of entropy). Sandbox keys (brubru_test_) hit the same endpoints against a sandbox meter for free integration testing.
• Storage: api_keys table with key_hash, key_prefix, scopes, last_used_at, revoked_at.
• Rate limit: per-key sliding window. Exceeded requests return 429 with a Retry-After header.
• Metered billing (Stripe, live): each call resolves a per-endpoint cost and debits the key holder's API-credit balance; usage is recorded per event and refunded automatically on upstream failure. Brubru's own operator key is billing-exempt so the company never pays to call its own API.

Brubru administrators mint, list, and revoke keys on behalf of Professional subscribers.

• POST /api/admin/users/{user_id}/api-keys: mint a key. Plaintext returned once, never again.
• GET /api/admin/users/{user_id}/api-keys: list a user's keys (active and revoked). Plaintext is never returned.
• DELETE /api/admin/api-keys/{key_id}: revoke a key by setting revoked_at. Revocation is an audit event, not a deletion.

Every paginated v1 endpoint returns the same shape so partner integrations are predictable across datasets.

• Pagination: total, pages, page, limit, has_more, next_page, remaining_pages.
• Filter echo: published_from, published_to, updated_from, updated_to, detail_level.
• Attribution: meta.powered_by = "Brubru" and meta.source = "brubru.beresol.eu": machine-readable so partners can display credit programmatically.
• Dataset-specific aggregates (e.g. by_country, by_user_type) sit in an aggregates object alongside data so partners get analytics and detail in one call.

v1 is the original hand-built core that the chatbot still runs on internally. v2 (above) generalised it to the full 934-endpoint surface with the uniform five-datapoint contract; v1 remains live for the partners who integrated against it. The v1 core:

Meta:

• GET /api/v1/ping: unauthenticated health check.
• GET /api/v1/whoami: returns the key holder's tier plus current rate-limit window.

Datasets:

• GET /api/v1/laws: 8,710 distinct EU laws (28,513 OJ entries, 61,219 XML files). Filters: celex, doc_type, policy_area, q (TSVECTOR full-text), published_from, published_to.
• GET /api/v1/procedures: 1,200+ legislative files. Filters: reference, committee, rapporteur_mep_id, status, updated_from, updated_to.
• GET /api/v1/commissioners/{name}/agenda: live calendar items for all 27 college members. Accent-insensitive name resolution.
• GET /api/v1/consultations/by-initiative/{id}/feedback: live stakeholder feedback from Have Your Say with country and user-type aggregates.

Legal-Text Intelligence:

• GET /api/v1/legal-text/{celex}/recital-article-map: TF-IDF cosine mapping, top-3 recitals per article.
• GET /api/v1/legal-text/{celex}/defined-terms: Article 3/4-style definitions dictionary.
• POST /api/v1/legal-text/resolve-references: plain text in, structured CELEX citations or annotated HTML out.
• POST /api/v1/legal-text/resolve-aliases: resolve human names (GDPR, DSA, AI Act, CBAM, ...) to CELEX. 680+ aliases.

Every Brubru Chat query flows through three retrieval tiers before the AI composes an answer. The v1 API is not only an external partner surface, it is the internal reinforcer that guarantees coverage.

• Tier 1, DB + smart filters. context_builder.py queries PostgreSQL directly: 8,710 distinct EU laws (28,513 OJ entries, 61,219 XML files), 4,156 calendar events, 564 knowledge guides, 3,266 legislative carriages, 145 EPRS publications, 24 committee work registers, 75 committee minutes, plus the 10 specialised EC clusters (119,542 rows). Fast, curated, no external dependency.
• Tier 2, on-demand live fetchers. Six specialised clients fire when intent detectors match: Doceo (plenary debates via CRE XML), EP Multimedia (committee meeting transcripts via on-demand Whisper transcription), Have Your Say API (stakeholder feedback), commission.europa.eu scrape (commissioner agendas), EUR-Lex Cellar (CELEX document fetch), Tavily (25 trusted .europa.eu + policy-media domains).
• Tier 3, v1 API backstop. When Tiers 1 and 2 leave a surface thin, the same service functions that power /api/v1/calendar/events and sibling endpoints re-query with broader filters. The simpler filter logic often catches what the chat's smart filters miss. Real example: on 17 April 2026 the "energy events next week?" query deflected because Tier 1 over-filtered; the Tier-3 backstop added 8 more relevant events. Promise: no query fails for lack of trying.

Phase B-minimal shipped 20 April 2026 (calendar backstop). Assessment checkpoint 4 May 2026, if materially improves answer quality, expand to guides, laws, EPRS, procedures, committees, MEPs.

The chatbot, My EU Bubble, Amendator, and Document Generator inside the Brubru app all consume the same /api/v1/* surface that paying customers do. This dogfooding rule means the API never lags behind the product, if a Brubru feature works, the API works. External partners get the same data quality Brubru itself relies on.

Today (May 2026): 47 v1 routers covering the chatbot's full context cascade, /knowledge-guides (509 guides), /laws/{celex}/text (full EU law text via Formex or EUR-Lex Cellar fallback), /eprs (145 EPRS publications), /committees/{code}/work-items + /minutes, /calendar/events (417 institutional events), /meps (live from EP Open Data), /predictions/{ref}/timeline + /outcome (XGBoost + baseline), plus the 10 specialised EC clusters and the 9-identifier resolver. The chatbot consumes the same surface external partners do.

Regression test: /training api. 50 representative user queries (EN/ES/CA/FR) from 7 query patterns are replayed against the live v1 API weekly. First run: 48/50 pass (96%). Every v1 change is verified against this baseline before ship. Dataset: data/training/api_queries_50.json.

Phase 2 (June 2026): cursor pagination, field projection (?fields=), sorting, ETag caching, X-RateLimit-Reset header across every endpoint. Phase 3 (September 2026): webhooks, bulk CSV/Parquet exports, multilingual responses (24 EU languages), semantic search via embeddings.

Brubru speaks the same vocabulary as the EU Publications Office. As of 2 May 2026, every authority code surfaced by the chatbot or API resolves to a multilingual label drawn from the official EU Authority Tables, every legal identifier is verified before delivery, and Brubru itself is published as a DCAT-AP-conformant dataset.

• Authority label cache. Local cache of the 12 highest-traffic EU Authority Tables (corporate bodies, procedures, document directories, modification types, and more), stored bilingually for the six languages Brubru supports. Refreshed nightly. The chatbot prepends a localised glossary block to its working context whenever an authority URI is in scope, the model sees both the URI and the human label in the user's language at the same time.
• Standard-identifier resolver. A unified resolver covering nine EU identifier families, CELEX, ELI, ECLI, EuroVoc URI, NAL URI, OJ reference, ISBN, ISSN, DOI, catalogue number. Drop in any free text and get back the structured identifiers, ready for downstream linking. Court rulings (ECLI) get a dedicated discover endpoint with deep-links to e-Justice.
• DCAT-AP self-description. Brubru's own datasets, the Catalan EU legislation library, the chat knowledge base, the v1 API surface, the EU laws TSVECTOR corpus, are described in the same DCAT-AP profile that data.europa.eu harvesters expect. Available as Turtle, RDF/XML, and JSON-LD. The pointer Europe's official data portal needs to discover us, structurally compatible with their existing pipelines.
• OP Core conformance on every response. Each v1 response carries Dublin-Core-aliased metadata (title, identifier, type, language, publisher, conforms-to) and an X-OP-Core-Conformance response header. Partners get standards-shaped responses out of the box, no custom adapter required.
• ELI + CDM ontology pinning. The legal-content ontologies that anchor Brubru's data (European Legislation Identifier and the Publications Office Common Data Model) are vendored at known versions, with a CI test that fails the build if a new predicate enters the codebase without a snapshot refresh. The ontology never silently shifts under our partners.
• Citation verification, extended. The model's outputs are verified against Cellar (for CELEX/COM) and against the local authority cache (for body codes and other authority URIs) before delivery. A class of "the AI cited a thing that doesn't exist" failures, common in generic GenAI tools applied to EU policy, is closed at the response layer.

Reference: docs/applications/euvoc.md (canonical living plan, 13 phases, status board + update log). Self-monitoring is built in: three free piggybacks on the daily news routine plus two new Friday training sub-commands (/training euvoc, /training drift) keep the layer healthy without adding manual work.

Brubru Chat is itself an AI system under EU Regulation 2024/1689 (the AI Act). We treat our own compliance as a product feature, not a legal afterthought. The canonical academic reference on agentic-system compliance is Nannini et al. April 2026, AI Agents Under EU Law: A Compliance Architecture for AI Providers, arXiv 2604.04604v1. The paper identifies four agent-specific compliance challenges (cybersecurity, oversight, transparency, behavioural drift) and nine parallel EU instruments that apply concurrently. What follows maps each against Brubru's architecture.

• Risk classification. Brubru Chat is not a high-risk AI system under Annex III: it does not perform CV screening, credit scoring, access to essential services, critical infrastructure control, or biometric identification. It falls under Article 50 transparency (general-purpose interaction with EU policy text) and Chapter V GPAI obligations as a deployer of foundation models.
• Transparency (Article 50). The paper identifies Article 50's "affected parties" requirement as a non-trivial engineering challenge. This entire /data-architecture page is our inventory: 519 europa.eu sources, the 3-tier retrieval cascade, every on-demand fetcher (plenary debates, committee transcripts, stakeholder feedback, commissioner agendas, Cellar, Tavily), every downstream user in scope (registered users, API partners, daily-brief recipients). Users interacting with Brubru Chat see the AI disclaimer; the daily brief is opt-in; partner API calls are logged per-key.
• Behavioural drift. The paper's taxonomy: anticipated adaptive behaviour (tool selection, in-context learning, RAG) is not substantial modification if documented and risk-assessed. Brubru's tool-calling pattern (intent detection -> DB smart filters -> on-demand fetchers -> v1 API backstop) is exactly this category. Every fetcher is documented in code and in this page. We do not perform continuous post-deployment weight updates; the Mistral / Claude / OpenAI / Gemini foundation models are externally versioned.
• Cybersecurity, Article 15(4). API-level least-privilege rather than system-prompt controls. Our v1 API ships with: scope-limited API keys (tied to subscription_tier, not to the user's full Brubru session), per-key sliding-window rate limits (60 req/min default), SHA-256 hashing of keys at rest, plaintext shown once at creation, structured error envelope with reason_code for diagnostics, and tier gating enforced per endpoint. No prompt-injection vector reaches the keys.
• Human oversight, Article 14. The /audit-queries skill pulls every user query from the DB and diagnoses each against a quality checklist, with root-cause tracing into context_builder.py. Since 20 April 2026 this routine explicitly includes a dogfooding comparison to the v1 API (Tier-3 backstop). See memory/feedback_chat_quality_paramount.md and memory/feedback_three_tier_coverage.md.
• Data governance, Article 10. Training data for foundation models is governed by their providers (Mistral, Anthropic, OpenAI, Google). Brubru's own knowledge base (509 guides, 11,393 multilingual triggers) is human-authored and version-controlled in git; every update is traceable. User chat logs are retained in line with GDPR + our subprocessor agreement (see /subprocessors).
• The 9-instrument compliance surface. Brubru itself also monitors all nine: AI Act, GDPR, CRA, DSA, Data Act, ePrivacy, NIS2, sector-specific (MDR, MiFID II, PSD2), revised Product Liability Directive. We ship daily updates to our knowledge base so customers with meaningful regulatory exposure (Enterprise-tier profile: Discord, Coinbase, Roblox, Epic Games, etc.) track the whole stack through a single surface rather than through nine parallel subscriptions.

Status: self-assessment based on the canonical academic source. Harmonised standards under M/613 (CEN/CENELEC JTC 21) remain working drafts at April 2026; when adopted, we will re-map our posture against published standards. Full source reference: Nannini, Leon Smith, Maggini, Panai, Feliciano, Tiulkanov, Maran, Gealy, Bisconti, April 2026.

• /api: public landing page (endpoints summary, response envelope, code samples, CTAs to docs and subscription). Header nav entry. No pricing shown here any more (pricing lives on /subscription).
• /api/docs/: full docs microsite shipped 20 April 2026: 10 written pages (~130 KB), sticky sidebar nav, responsive. Introduction, Authentication, Envelope, Errors, Rate limits, Endpoints (the full v2 catalogue), Institutions matrix, Glossary (47 EU policy terms in plain English), Use cases (6 personas + 4 competitor patterns), Changelog. Soft-paywall banner links non-Professional visitors to /subscription but doesn't block reading.
• /api/v1/openapi.json: OpenAPI 3.1 spec auto-generated by FastAPI, filtered to v1 only. Postman imports in one click.
• Scalar live try-it reference remains at brubru-production.up.railway.app/api/docs (backend-served, execution requires API key).

The same surface is wrapped by two official Python libraries on PyPI (brubru and brubru-eurovoc, open source at github.com/Beresol-BV/brubru-EU-scraper-library), so a data team is one import from the whole estate.

• brubru: a thin SDK over the v2 API. Typed objects over the five-datapoint contract, automatic pagination, honest errors. The API's own client. bru.extract(url, classify=True), bru.social.posts(...).
• eurovoc: a standalone, open-source EuroVoc classifier. It tags any text into the EU's official subject space (7,029 descriptors, 127 microthesauri, 21 domains), multilingual including Catalan. A modern, open successor to the 2021 PyEuroVoc package, and the natural home for a retrained long-context model. Runs locally or, with backend="brubru", through the hosted classifier.

Documented on /api and /api/docs/.

Beyond what institutions publish, Brubru maps what EU actors say on social media: the layer that often precedes or explains the formal documents. A polymorphic directory holds 3,800+ mapped accounts across institutions, agencies, MEPs, Commissioners, EU officials and EU-affairs journalists, spanning every major platform. Content is fetched only where it is free and terms-clean (Bluesky, Mastodon, YouTube, and X via the public syndication endpoint); Instagram, LinkedIn and TikTok are mapping-only by design (no paid API, no covert scraping).

The whole layer is exposed under /api/v2/social with the same five-datapoint contract: the directory overview, posts (newest first, filterable by entity and platform), accounts, a per-entity social footprint, and the platform pick-list with each platform's content-fetch tier.

• Day 1 (15 April 2026), API key model, migration 033, admin endpoints, six-path auth dependency. 10 tests.
• Day 2 (15 April 2026), /api/v1/* sub-router, canonical paginated envelope, per-key sliding-window rate limiter (60 req/min), X-RateLimit-* headers, ping/whoami/feedback endpoints. 25 tests.
• Day 3 (15 April 2026), /api/v1/laws and /api/v1/procedures shipped with full filter sets. 30 tests.
• Day 4 (15 April 2026), /api/v1/commissioners/{name}/agenda shipped, four legal-text intelligence endpoints ported to v1. 34 tests.
• Day 5 (15 April 2026), Scalar docs at /api/docs, filtered OpenAPI at /api/v1/openapi.json, public landing page at /api, header nav entry next to Tenderator (i18n in 6 languages). 36 tests passing in total.
• Landing-page integration: the "For developers and RegTech partners" section now leads with the REST API (six flagship endpoints) plus the existing MCP server (six tools), with CTAs to /api and /api/docs.
• Jordi feedback pass (20 April 2026), 11 correctness bugs fixed after the 18 April integration review:
  • /laws: 422 conflicting_params when published_to and published_end carry different values; 422 invalid_date_range on inverted ranges.
  • /meps: rewired to direct httpx against EP Open Data REST v2. full_name and profile_url always populated; total now offset + items + 1-hint with coverage_complete: false.
  • /consultations: ISO-8601 dates (YYYY-MM-DDTHH:MM:SSZ); feedback_id always string.
  • /legal-text/{celex}/recital-article-map: graceful 503 computation_unavailable instead of uncaught 500.
  • /legal-text/resolve-references + /resolve-aliases: documented as READ-ONLY idempotent.
  • /commissioners/{name}/agenda: detail_url parser prefers title anchors; rejects #, javascript:, mailto:.
  • Default limit 20 → 50 across all 9 paginated endpoints.
  • 8 regression tests in test_v1_jordi_feedback.py; 68 total green.
• Day 6 (15 April 2026), partner-compatibility pass inspired by GovClipping's Postman docs. Six items shipped:
  • Structured error envelope on every /api/v1/* response: { "error", "reason_code", "request_id" }. Reason codes are stable machine-readable strings (auth_missing_key, auth_invalid_format, auth_key_not_found, tier_insufficient, rate_limited, not_found, invalid_query, upstream_error).
  • X-Request-Id response header on every call (echoes the client's inbound value if supplied, otherwise generates a UUID4). Matches the request_id in error bodies for support traceability.
  • Envelope additions: returned (items in this page) and coverage_complete (True for DB-backed, False for live-scraped results).
  • Filter aliases: published_end and updated_end accepted alongside published_to / updated_to. The envelope echoes both. Matches GovClipping's convention so partners reuse existing SDK patterns.
  • Dual authentication: Authorization: Bearer <key> (OpenAPI/Postman default) and X-API-Key: <key> (curl-friendly) both accepted. Bearer wins if both present.
  • GET /api/v1/meta/enums: a single lookup endpoint returning every valid filter value, doc_types, policy_areas (live from DB), EP committees, procedure statuses, stakeholder user types, detail levels. Partners call once, cache forever. 49 tests passing in total.
• Internationalisation: the /api landing page is now translated to all six Brubru languages (English, Spanish, Catalan, French, Italian, Dutch) via the api i18n namespace in each locale file. Fully responsive (desktop / tablet / mobile breakpoints).

Caveat: rate limiter is in-memory (per backend instance). For one paying partner this is fine; pin Railway to one replica until Redis is added.

Shipping the next-scale source coverage without building 500 bespoke scrapers. The pattern: one generic RSS ingester + YAML config per feed. Adding a new EU institutional source is now a config file, not code.

• Model + migration 035: institutional_publications table with unified shape (source_slug, institution_slug, category, title, summary, url, policy_areas, tags, published_date, full-text index). Row-Level Security enabled.
• Ingester: backend/services/ingestion/rss_ingestor.py reads YAML configs from configs/*.yaml, fetches via BaseRSSClient, upserts with dedupe on (source_slug, external_id).
• Catalog (discovery-driven): backend/scripts/discover_feeds.py crawls all 519 base domains from data/europa_eu/europa.eu_all_sources.md, probes each for <link rel="alternate" type="application/rss+xml"> in the HTML head plus 13 common paths. Validated URLs land in data/europa_eu/discovered_feeds.csv. configs_from_discovered.py then turns each CSV row into a YAML ingestion config automatically. First full 519-domain discovery: 64 validated feeds found, 54 auto-generated YAML configs.
• Live ingestion: 52 of 78 configured feeds working after the discovery sweep, 587 real publications across 44 sources and 35 institutions in the DB right now. Top contributors: JRC (91), European Commission (60), EMA (58), ECB (25), EEA (20), Frontex (20), EESC (20), CEPOL (20), FRA (12), EIGE (12). The remaining 26 configured URLs return 404/403 (retired feeds or WAF-blocked) and will be retired or re-curated in the next discovery pass.
• CLI: python3.12 scripts/ingest_publications.py [slug1 slug2 ...]. Run for all configs or a subset.
• New API: GET /api/v1/publications with filters (q, institution_slug, source_slug, category, policy_area, published_from/to). Plus GET /api/v1/publications/sources listing available feeds with counts. Same canonical envelope as every v1 endpoint.
• Tests: 58/58 passing (9 new tests for publications + ingester added).

Ongoing curation: each week new YAML configs land. When an upstream feed URL breaks, the ingester logs the error and continues with the others, no cascading failure. This scales without rebuilding code.

The same data that powers the REST API is exposed over Anthropic's Model Context Protocol so partner AI assistants (Claude Desktop, Claude Code, third-party MCP clients) can query Brubru natively. Six tools live today, mirroring the six v1 surfaces partners ask for most.

Same auth model as REST (API key in X-API-Key or Bearer). MCP positioning is significant: Anthropic and other AI vendors are pushing MCP as the agent-to-data standard for 2026. Brubru being first-class on MCP makes the chatbot, the API, and external AI agents share one truth layer.

The v2 API has moved coverage from a handful of streams to the full institutional map: 934 endpoints drawing on roughly 40 official EU machine-readable APIs unified under one contract, plus around 50 sources Brubru reverse-engineered or scrapes itself because they ship no API. data/europa_eu/europa.eu_all_sources.md remains the source universe (519 europa.eu domains tracked); the API now surfaces the institutionally important core of it, not a thin sample.

The remaining frontier is breadth on the long tail. The generic RSS ingestion pipeline (institutional_publications + automated feed discovery via discover_feeds.py → ingest_publications.py) lets a new feed be added as a YAML config rather than code, so coverage grows on a weekly cadence without bespoke scrapers. Expansion of the long tail stays a tiered programme (memory/project_data_provider.md): structured APIs first, then RSS/Atom feeds, then generic HTML, with Tavily as the real-time backstop. Partner-driven priority: deepen the sources paying customers actually pull, before chasing the tail.

• Full  81 bodies have a dedicated /api/v2 folder with the full agency pattern. The premium tier.
• Light  17 bodies are already pulled in, but only as an RSS feed or a news / events scraper, with no dedicated folder. These are the promotion backlog.
• Absent  15 bodies have no coverage in any tier yet. These are the build-from-zero backlog, almost all Joint Undertakings and the defence cluster.

Columns below: Folder = dedicated /api/v2/<body> surface; RSS = feed in /api/v1/publications; News = news / events scraper. Audit date 25 June 2026.

Build from zero (0 Absent): the build-from-zero backlog is cleared. Every body in this audit now has a dedicated folder or is a documented Light feed pending promotion.

Promote to a folder (9 Light): bodies already pulled in via RSS or a news scraper but without the full agency pattern, e.g. EESC, CoR and the EPPO. Promotion upgrades them from a single feed to the structured 5-datapoint surface.

Goal: drive all ~90 bodies to Full, so the API can claim, literally and verifiably, every EU institution, body and agency.

Since ~2018 the Commission has pushed two things to standardise the estate: the Europa Component Library (ECL), a mandated design system, and OpenEuropa, an open-source Drupal distribution. Most europa.eu sites have converged on them, predictable, server-rendered, scrapable.

But the mandate has gaps. Older institutions predate it (the Court of Justice runs a French enterprise CMS); some standardised on Microsoft (the Court of Auditors on SharePoint); the newer, independently-procured Joint Undertakings reached for WordPress or Microsoft Power Pages. And a rising anti-bot layer (Cloudflare-style challenges, proof-of-work walls) is being bolted on top of all of them, aimed squarely at AI scrapers. Brubru's value is having mapped and cracked each type.

What: the Commission's official front-end design system (europa.eu/component-library), mandated across europa.eu domains. How: pages are assembled server-side from standard components; content items carry predictable classes (.ecl-content-item) and URLs end in _en. Why: one accessible (WCAG 2.1 AA), multilingual, branded look across 300+ sites, centrally governed. Brubru technique: parse the .ecl-content-item list server-side, no browser needed. The easiest type.

Bodies: most EC Directorates-General, plus CINEA, AMLA, EACEA, ECCC, EIOPA, EISMEA, EPPO, EPSO, ERCEA, ESDC, HaDEA, REA, the European School of Administration, and the ECL Joint Undertakings (Clean Hydrogen, Global Health EDCTP3, EuroHPC, Euratom Supply Agency).

What: Drupal, usually the Commission's open-source "OpenEuropa" distribution (a Drupal profile + ECL theme maintained by DG DIGIT). How: content lives as "nodes" rendered in .teaser / .card / .views-row / article.node / .group-content templates, paginated with ?page=N. Why: open-source, EC-backed, gives agencies the ECL look with editorial freedom, the default for decentralised agencies. Brubru technique: a generic teaser-walker handles the common case; per-agency overrides where the node template diverges.

Bodies: ACER, BEREC, CdT, CPVO, EASA, ECDC, EDPB, EDPS, EESC, EFCA, EFSA, EIT, ELA, EMA, ENISA, ERA, ESM, ESMA, ETF, EU-OSHA, EUISS, eu-LISA, Eurojust, Europol, EUSPA, SESAR, SRB, CBE.

What: WordPress (often with the Elementor visual page-builder), the world's most common CMS. How: bespoke themes; news at /latest-news/<slug>; documents frequently as PDFs whose only title is the filename. Why: the JUs are small, recently created and independently procured, WordPress is cheap, fast and familiar. The trade-off: every JU's markup is different. Brubru technique: per-JU parsers (filename-to-title extraction, Elementor post-grids rendered through a headless browser).

Bodies: Fusion for Energy (F4E), Europe's Rail (Elementor), Smart Networks & Services (SNS).

What: a Microsoft Dynamics 365 / Power Pages portal (low-code, Dataverse-backed). How: content lives in Dataverse and is rendered client-side via AJAX; pages are addressed by query-string IDs (/NewsDetails?id=…). Why: some bodies standardise on the Microsoft enterprise stack. Brubru technique: a headless browser plus the ?id= detail pattern, there are no server-rendered cards and dates load late, so this is one of the harder types.

Bodies: Chips JU.

What: Jalios JCMS, a French enterprise Java CMS/portal. How: distinctive jcms/… URL paths and its own markup conventions. Why: the Court (Luxembourg) procured independently, long before ECL existed, and chose a European enterprise CMS. Brubru technique: JCMS-specific path + selector handling.

Bodies: Court of Justice of the EU (curia.europa.eu).

What: Microsoft SharePoint serving public pages. How: an enterprise content/collaboration platform; heavy, JavaScript-driven, with awkward .ms-rte* markup. Why: the institution standardised on Microsoft 365. Brubru technique: a headless browser with tolerant selectors.

Bodies: European Court of Auditors (ECA).

What: a Single Page Application, the browser builds the page in JavaScript from API calls; the served HTML is near-empty. How: a JS framework (React/Angular) over a search service (EUIPO is fronted by Algolia; EUR-Lex by a JavaScript-gated search). Why: rich interactive UX, instant faceted search. The worst case for extraction: you must run a headless browser or reverse-engineer the underlying JSON/Algolia endpoint. Brubru technique: hit the underlying API where one exists, else render in Playwright.

Bodies: EUIPO (Algolia anti-bot SPA), EUR-Lex (JS search).

What: hand-built sites with custom article markup (.post-meta, .date-details, .event-dates). Why: built before or outside the ECL mandate, often by an external agency, so no shared template applies. Brubru technique: a per-site selector map, maintained by hand.

Bodies: EDA, CoR, ENISA, ACER (partly), SESAR (date fields).

Any competitor can scrape one ECL site. The moat is the long tail: knowing, for all ~90 bodies, exactly which of these eight platforms and seven walls applies, and carrying a tested extractor for each. Brubru encodes that map as code (one scraper module per body, the platform type documented in its header) and as this living reference. It is the difference between "we fetch some EU news" and "we reliably extract structured data from every EU institution, body and agency, whatever they built it on."

Source: every backend/services/scrapers/economy_*.py module, whose docstring records the platform type and extraction approach. Mapped 26 June 2026.

8,544 URLs · 232 domains · 87% reachable (HTTP 200) · 0 mojibake titles after the encoding fix · junk titles near-zero.

Platform mix (reachable HTML): Drupal 4,068 · ECL 2,808 · SPA 285 · WordPress 75 · JCMS 48 · SharePoint 20 · Dynamics 14. Two platforms are ~80% of the estate, so two handlers cover most of it; the long tail is anti-bot walls (europarl, FRA, EUDA) and fully-JS SPAs.

Caveat: the 28-worker concurrent scan trips some per-host rate limits, so 429/403 counts are an upper bound on "blocked". Scanned 27 June 2026.

Mojibake ("Europe’s"): requests defaulted an undeclared charset to Latin-1, but EU pages are UTF-8 → honour the real encoding. 0 mojibake estate-wide.

Nav / CTA junk as items ("Subscribe", "Read more", "Registration") → a central junk-title filter at every item-build site.

Dates near-zero: the parser missed ordinals ("24th June"), US and numeric formats → a format-robust parser preferring real date elements; plus an opt-in detail-page complete_dates for sites that only show the date on the article page (eige 0→88%, easa 0→100%).

Forms as items: filter widgets and date-pickers were extracted as dateless "items" → strip form / .ecl-datepicker before card selection.

Proper-noun pollution ("Chips JU" → "jute"): a confidence gate keeps a tag only when top matches agree on a EuroVoc domain or the margin is clear. No tags beats wrong tags.

Organisation / treaty NAMES as subjects ("Pacific Islands Forum", "EAEC Treaty"): filtered using EuroVoc's own microthesaurus domains, while keeping geography and legal subjects.

Thin titles: an opt-in deep mode classifies on the full article body instead of the listing snippet.

The chips-ju data-grid renders ~12s late and hides links behind JS "Click here" spans, so it returned 0 items. Fixed with a render-settle wait (poll until content stops growing and no spinner shows), a permissive reader for its heading+span card layout, and a one-shot browser retry. 0 → 6 items, now reliable. A regression this caused on the Court of Justice (curia grabbing menu chrome) was caught and fixed by scoping the permissive path, the lesson being that every relaxation needs a cross-platform regression sweep.

Hard anti-bot walls (EUDA 403, FRA Anubis) stay blocked. A few bodies (EUISS, "Conference on Disarmament") are filed by EuroVoc under topic domains, so they can survive the org-name filter on security text. A small class of fully-JS sites (enisa) hide the date in relative or image form, unrecoverable even when rendered.

Source: scripts/extract_estate_scan.py + extract_engine_issues.md. Engine: backend/services/extract/.