LibraryUniverse Graph
In this page
Deep technical documentation for universe relationships, graph modeling, and cross-media linking.
Where this lives in the code
src/MediaEngine.Domain/Contracts/IUniverseGraphQueryService.cssrc/MediaEngine.Providerssrc/MediaEngine.Api/Services/Plugins/PluginUniverseLoreService.cssrc/MediaEngine.Web/Components/Universe
Purpose
The Universe Graph connects fictional characters, locations, organizations, and events across all media in the Library. When a book, film, audiobook, and comic all belong to the same creative universe, the graph captures the relationships that bind them - characters who are siblings, locations that exist within other locations, actors who played specific roles across adaptations.
The graph is populated automatically during hydration using Wikidata as the canonical data source. Approved plugins can add supplemental lore, but plugin data stays marked as supplemental and keeps source attribution instead of becoming canonical Wikidata data. It is stored in SQLite and loaded into an in-memory EntityGraph for graph queries and pathfinding.
Current serving rule: graph APIs and UI must remain truthful. GET /universes returns narrative roots with real entity and relationship counts plus an enrichment status. Chronicle Explorer loads graph data lazily only when opened, renders a graph only when real nodes and edges exist, keeps the explorer usable on phones (shorter full-width graph, touch zoom and pan, a node list under it), and disables the timeline control on television devices. Empty, partial, and pending graph states are surfaced as data lifecycle states rather than replaced with fake entities or links.
Supplemental serving rule: approved plugin lore appears only when a caller asks for the overlay with include_supplemental_lore=true. Plugin nodes and edges include supplemental=true, provenance=plugin, source_plugin, and source_url so UI and API clients can distinguish them from Wikidata facts.
Universe page Connections (/universe/{qid}/overview, built by UniverseConnectionsReadService) are evidence only and name a source on every row. A Character, Cast credit or adaptation’s source is the provenance stored on the appearance link or relationship (Wikidata today); an approved plugin lore source that holds the same Wikidata item is added beside it. Characters come only from Character-type entities linked to a work the profile may see. Visibility of a link is one set for every Universe read, OwnedStory.VisibleQids: the identity of each work the profile may see plus the identity of the show (root work) each belongs to, so a link to a show counts whenever any of its episodes is visible, even when every episode carries an identity of its own. Characters are judged by one rule, SpoilerRule in the Domain (spec U-WP11), called through CharacterSpoilerService by every read that names a character: the Universe page Characters and Cast, the compact /api/v1/universe/{qid} read, /universe/{qid}/cast, “Also appears in” and the graph, paths and family-tree actions (a hidden character is omitted from the graph with every edge to it).
First seen at (computed, nothing stored). CharacterFirstSeenReader / OwnedStory turn a character’s appearance links into one position per title it appears in, from the authorised display projection of the requesting profile. A link’s work QID that is an owned episode’s own identity (a child Work carrying its own wikidata_qid) is that episode: show Work, season number (a missing season reads as 1) and episode number. Among a title’s episode links the earliest wins, and a first_appearance link wins over a plain presence link. A link that names the show, a film or a book as a whole is that title with the position within it unknown; books and audiobooks never get a chapter because no source provides one. Matching is by Wikidata item only: TMDB per-episode character names have no QIDs and are never used to place anyone.
The rule. Inputs: the profile’s spoiler level (up_to_progress default, or everything, stored in the profile’s playback settings and changed only through PUT /profiles/{id}/settings/spoilers), the character’s first-seen positions, the “spoiler for” (P7528) works and the profile’s progress (user_states, read through IViewerProgressRepository: a title is started once any file has progress above 0 and finished once every owned file is at or above 99.5; a show’s furthest started and furthest finished episodes are kept). Output per character: Shown, NoPlotDetails (named with its source, no description, no work count; “Also appears in” still opens, because it lists Works, not plot) or Hidden (omitted, counted in hidden_character_count). At everything every character is Shown. Otherwise a spoiler marker for a Work the profile has not finished or does not own makes it Hidden (the marker counts only when every visible appearance carries one; a character with a plain appearance is known from that); then the most permissive title wins (a character met in any one title is Shown, even if another title first sees them later; a title the profile has finished shows everyone in it, whether or not an episode is known for them): an episode at or before the furthest started episode is Shown, a finished title is Shown, a title not started or a position unknown within a started title is NoPlotDetails, and an episode later than the furthest started one is Hidden. No profile is judged as a viewer who has started nothing. Character performer credits (character_performer_links) store no provenance of their own; they are written only from Wikidata cast statements, so a Cast row takes its source from the matching appearance link.
Narrative Root Resolution
Before the graph can be built, the Engine determines which Universe a work belongs to. There is one Universe record per Wikidata ID: the narrative_roots row. The Universe shelf the Library shows (collections.universe_qid on a Universe collection) and the graph root are the same record, chosen by one rule, UniverseIdentityRule (Domain), which both ParentCollectionResolver (Series shelves) and NarrativeRootResolver (works) use:
- P1434 (takes place in fictional universe)
- P8345 (media franchise)
- P179 (part of series) naming a broader Series than the work’s or shelf’s own
- P144 (based on), so an adaptation joins the Universe of what it adapts
- the work’s or shelf’s own Series (P179)
Within one step the lowest QID wins, so repeated runs pick the same record. A fictional universe or franchise value that is one of the work’s own characters, places, events or objects is ignored. The chosen QID is then looked up in the Library: a Series shelf with that QID that already names a Universe gives that Universe, otherwise the record’s parent_qid chain is followed to its top. A work that sits on a Series shelf takes that shelf’s Universe; when a shelf’s Universe changes, the narrative root stored on its works follows, and shelves that point at it (a TV show based on that book Series) are decided again, so the result does not depend on which shelf was ingested first.
The resolved QID and label are stored on the work (narrative_root_qid, narrative_root_label). All fictional entities discovered within a work are linked to its narrative root, enabling cross-media queries - for example, finding all characters from the Dune universe across novels, films, and audiobooks. Entities discovered under an earlier root keep it until the next universe enrichment pass.
Entity Discovery
RecursiveFictionalEntityService handles entity creation and enrichment:
- For each entity QID discovered in a work’s Wikidata properties, finds or creates a
FictionalEntityrecord (keyed bywikidata_qid, UNIQUE constraint) - Links the entity to the work via
fictional_entity_work_links - If the entity has not yet been enriched (
enriched_at IS NULL), enqueues a Data Extension request for its properties - After enrichment,
RelationshipPopulationServicereads statement-level graph evidence and creates graph edges and work appearances. It uses normalized_qidclaims only as a compatibility fallback when structured evidence is unavailable.
Entity types are stored in a single fictional_entities table with a sub-type discriminator:
| Type | Examples |
|---|---|
| Character | Paul Atreides, Frodo Baggins, Ellen Ripley |
| Location | Arrakis, Mordor, Nostromo |
| Organization | House Atreides, The Fellowship, Weyland-Yutani |
| Event | Harkonnen assault on Arrakis, Battle of Helm’s Deep |
| Object | Artifacts, vehicles, weapons, devices, and important in-fiction documents |
Relationship Types
RelationshipPopulationService creates graph edges by reading entity-valued claims after enrichment. The traversal depth is configurable (default: 2 hops) via lineage_depth in config/hydration.json.
27 character, place, organisation and event relationship types are supported. A property is read only when it is listed for its entity type in data_extension (character_properties.core for characters) and has a property_labels entry in config/providers/wikidata_reconciliation.json; an edge is written only for the claim keys RelationshipPopulationService maps to a type:
| Relationship | Wikidata property |
|---|---|
| father | P22 |
| mother | P25 |
| spouse | P26 |
| sibling | P3373 |
| child | P40 |
| participant_in (a character took part in an event) | P1344 |
| enemy | P7047 |
| student_of | P1066 |
| partner | P451 |
| member_of | P463 |
| allegiance | P945 |
| educated_at | P69 |
| residence | P551 |
| creator | P170 |
| located_in | P131 |
| part_of | P361 |
| chief_executive_officer | P169 |
| parent_organization | P749 |
| has_parts | P527 |
| position_held | P39 |
| conflict | P607 |
| significant_person | P3342 |
| affiliation | P1416 |
| occurs_at | P276 |
| participant | P710 |
| caused_by | P828 |
| causes | P1542 |
Adaptation edges between Works
Adaptations on a Universe page fill from evidence. During Stage 3 (right after the Work’s narrative root is resolved) WorkAdaptationEdgeService reads the Work’s own Wikidata claims and writes entity_relationships rows between Works, so no adaptation is guessed:
| Claim on the Work | Wikidata property | Edge written |
|---|---|---|
based_on | P144 | this Work based_on the named Work |
derivative_work | P4969 | the named Work derivative_work of this Work (the claim is stored reversed, so every edge reads “subject is an adaptation of object”) |
An edge is keyed by subject, type, object and source, so a re-run updates the row it wrote before and never duplicates it. Works are not fictional_entities rows; their QIDs sit as plain text in the edge. Ownership is not checked when an edge is written. The Universe reads (UniverseConnectionsReadService) offer the profile’s visible Work identities (OwnedStory.VisibleQids) alongside the entities’ when they look for adaptation edges, and show an edge only when both of its Works are visible and one of them is the Universe’s own (a Series member, a standalone work, or an entity); each Work is named, typed and given its cover from the profile’s own display rows. The adaptation tree action anchors on the Works its characters appear in. A statement this service wrote carries a work-adaptation|... statement key, and is removed when its claim is gone, so an edge cannot outlive the claim behind it; edges another writer stored are never touched. The claims are Parent-scope: for an episode, the fallback identity is the show’s, never the episode’s. A failure writing edges is logged and does not stop the rest of stage 3. inspired_by (P941) is not read yet, so no inspired_by edge is written.
Actor-to-character links are stored separately in character_performer_links (person_id, fictional_entity_id, work_qid) and merged into the graph at query time.
EntityGraph In-Memory Graph
The UniverseGraphQueryService loads entities and relationships from SQLite into a Tuvima.Wikidata.Graph in-memory graph. The graph is lazy-loaded per universe and cached - it is not rebuilt on every query.
Capabilities served from the in-memory graph:
- Pathfinding between any two entities
- Family tree traversal
- Cross-media queries (all characters from universe X appearing in media type Y)
- Era-filtered queries (entities and relationships active at a given timeline year)
Person Infrastructure
Persons (authors, directors, narrators, actors) are distinct from fictional entities but participate in the same graph via performer links.
Each Person record carries:
- Core identity: name, WikidataQid, role (Author, Narrator, Director, Cast Member, Voice Actor, Screenwriter, Composer, Translator, Editor, Host, Producer, Illustrator)
- Biographical fields: date_of_birth, date_of_death, place_of_birth, place_of_death, nationality, is_pseudonym
- Social links stored as Actionable URI Schemes (see below)
- Pseudonym links via
person_aliasestable (bidirectional: P1773 attributed_to, P742 pseudonym)
Person headshots on disk: .data/assets/people/{personId}/headshot.* referenced by persons.local_headshot_path. Wikimedia Commons P18 is strictly Person-only - never used for media cover art.
Actionable URI Schemes
Social links are stored in a format that enables native app launching on mobile and automotive device profiles:
| Platform | Stored format | Web fallback |
|---|---|---|
instagram://user?username={handle} | https://instagram.com/{handle} | |
| Twitter/X | twitter://user?screen_name={handle} | https://x.com/{handle} |
| TikTok | tiktok://user?username={handle} | https://tiktok.com/@{handle} |
| Mastodon | https://{instance}/@{user} | Same |
| Website | Direct URL | Same |
DeviceContextService selects the appropriate URI format at render time: URI scheme for Mobile/Automotive, HTTPS for Web/Television.
Chronicle Engine
The Chronicle Engine extends the Universe Graph with time-awareness.
Temporal Qualifiers
Relationships carry StartTime and EndTime (nullable ISO 8601 strings) sourced from Wikidata P580/P582 temporal qualifiers. A character may be married for part of a story, a faction may exist only during a specific era, an actor may have played a role in one adaptation but not another. The fact also owns normalized queryable qualifiers, so work scope (P10663), point-in-time (P585), fictional time index (P4895), spoiler boundaries (P7528), statement nature, and provenance are not flattened or lost when the same graph triple is asserted in different contexts. Work appearances preserve the same context, along with link type/role and optional ordinal or scene anchors.
Lore Delta Detection
ILoreDeltaService batch-fetches current Wikidata revision IDs via wbgetentities?props=info and compares them against the stored WikidataRevisionId on each FictionalEntity. Each entity is reported as a LoreDeltaResult record and sent over the wire as the Contracts UniverseLoreDeltaResponse (entity_qid, label, cached_revision, current_revision, has_changed); the Dashboard counts the items with has_changed. The check runs when the Chronicle Explorer page loads (GET /universe/{qid}/lore-delta); when it reports changes, the Explorer shows a banner. The Engine does not push a live notification for lore changes.
Era-Correct Actor Resolution
IEraActorResolverService resolves which actor played a character for a given timeline year. It queries performer edges for the character, filters by temporal range, and returns the matching actor’s headshot URL. Falls back to the most recent performer when no temporal match exists.
Canon Discrepancy Detection
ICanonDiscrepancyService compares an edition’s canonical values against its master work (P629 edition_or_translation_of) across six core fields: title, author, year, genre, series, series_position. Discrepancies surface via GET /metadata/{entityId}/canon-discrepancies.
Chronicle Explorer
Route: /universe/{Qid}/explore
The Chronicle Explorer Dashboard page renders the full universe graph using Cytoscape.js (vendored at wwwroot/lib/cytoscape/cytoscape.min.js, MIT license). JS interop module at wwwroot/js/cytoscape-interop.js exposes: initGraph, updateGraph, filterByTimelineYear, focusNode, zoomIn, zoomOut, fit, resetView, setLayout, destroy.
Features:
- Universe header with entity and edge count chips
- Lore Delta amber alert banner when Wikidata changes are detected
- Sources tab for admin approval of supplemental lore sources
- Timeline slider (visible when edges carry temporal data)
- Type filter toggle chips (Character / Location / Organization)
- Layout selector (force-directed / concentric / grid)
- Cytoscape.js graph panel (60%) with node-click detail drawer
- Searchable entity list panel (40%): every node sorted by label with its type and relationship count, one real button per row that centres the node and opens the same detail drawer as a graph tap
- Zoom in, Zoom out, Fit and Reset buttons above the graph (the pinch-free path for remotes and keyboards)
Device constraints: on phones the graph is full width with a height of at most 70% of the viewport and the controls wrap; pinch-zoom and drag-pan are Cytoscape’s native touch gestures. The timeline slider is disabled on television.
Supplemental Lore Approval
The first implemented universe lore plugin is tuvima.fandom-lore. It reads Wikidata P6262 Fandom article IDs to suggest candidate Fandom wiki sources, or admins can add a Fandom URL manually. Approval happens in Chronicle Explorer’s Sources tab for the selected universe. Pending sources are inert; the plugin only imports entities or relationships from sources with status Approved.
What the lore plugin may send, and where:
- Only Universes the Library holds. Discover, add-a-source and enrich act on a QID only when it is a narrative root or a root Universe collection already in this Library (
ILibraryUniverseQidGate, checked once inPluginUniverseLoreServiceand in the Fandom discovery application operation). Any other QID answers 404 (a 400 invalid-payload answer for the application operation) and sends nothing out. - Only Wikidata and Fandom hosts. Wikidata is called only at
www.wikidata.org, with the Universe’s own QID (action=wbgetclaims&entity=Q…&property=P6262). A source’s address must behttpsonfandom.comor a subdomain such asstarwars.fandom.com(FandomHostRule). AP6262value that points anywhere else is ignored; an admin’s manually added address on another host is refused with a 400 (“Only Fandom wiki addresses can be added.”); an already stored source on another host is skipped with a log line and never called. The structured page request carries no title, author, file name or search term. - No title list leaves the machine. Work titles, authors and file names are never part of any lore request (
UniverseLorePrivacyTests.NoTitleListLeavesTheMachine). The User-Agent carries only the contact the admin chose to enter in the plugin settings.
The imported rows live in:
| Table | Purpose |
|---|---|
plugin_lore_sources | Candidate, approved, and rejected external lore sources per universe and plugin. |
plugin_lore_entities | Supplemental characters, locations, organizations, events, and objects with source attribution. |
plugin_lore_relationships | Supplemental relationships between approved plugin lore entities. |
Stage 3 universe enrichment runs plugin lore after the core Wikidata enhancer pass. Plugin failures are logged but do not block ingestion readiness.
API Endpoints
| Method | Route | Purpose |
|---|---|---|
| GET | /universes | List all narrative roots with entity counts |
| GET | /universe/{qid} | Universe summary: label, level, and counts of the entities (all, characters, locations, organizations) and relationships visible to the caller |
| GET | /api/v1/universe/{qid} | Paired-app Universe page: the Universe’s name, hero totals, member shelves (each with its “Also in this world” row) from the Collections catalogue, plus a read-only cast and adaptations list read through the same visibility rule as the graph (the shared UniverseConnectionsReadService), limited to what the paired profile may see; adaptations are empty when no adaptation relationship exists |
| GET | /universe/{qid}/graph | Cytoscape.js-ready JSON; supports ?types=, ?work=, ?center= with ?depth= (ego network), ?timeline_year=, and ?include_supplemental_lore=true filters |
| GET | /universe/{qid}/paths | Paths between two entities (?from=, ?to=, optional ?maxHops=, default 4) |
| GET | /universe/{qid}/family-tree | Family tree for one character (?character=, optional ?generations=, default 3) |
| GET | /universe/{qid}/cast | Characters with their real-world performers |
| GET | /universe/{qid}/adaptations | Adaptation tree built from based-on, derivative-work and inspired-by relationships, each child with its source; empty when none are recorded (never a flat list of every entity) |
| GET | /universe/{qid}/lore-delta | Check for Wikidata revisions newer than stored |
| GET | /universe/{qid}/lore-sources | Admin list of plugin lore sources for the universe |
| POST | /universe/{qid}/lore-sources/discover | Ask enabled lore plugins to suggest source candidates; 404 when the Library does not hold the Universe |
| POST | /universe/{qid}/lore-sources/manual | Add a manual source candidate; 404 when the Library does not hold the Universe, 400 when the address is not a Fandom wiki |
| POST | /universe/{qid}/lore-sources/{sourceId}/approve | Approve a source for plugin enrichment |
| POST | /universe/{qid}/lore-sources/{sourceId}/reject | Reject a source |
| POST | /universe/{qid}/lore/enrich | Import approved plugin lore for the universe; 404 when the Library does not hold the Universe |
| POST | /library/enrichment/universe/trigger | Queue a universe enrichment sweep (takes no QID) |
| GET | /library/portraits/{portraitId} | Stream one character portrait image (cast credits link to it) |
| GET | /metadata/{entityId}/canon-discrepancies | Compare edition against master work |
Configuration
Universe graph parameters in config/hydration.json:
Shared editor API
/entity-editor/universes/{qid} is the backend contract used by the shared
editor for graph targets; it does not introduce a second editor surface. A
Universe target is identified by its authoritative narrative_roots.qid. A
fictional-entity target carries both its internal GUID and Wikidata QID.
Only details and artwork capabilities are editable. Organization member and
Event participant projections are subtype-aware. Relationship,
appearance, in-universe timeline, source, history, and enrichment sections are
read or refresh projections. User detail edits are stored separately from
provider-enriched values and EntityAsset user overrides remain preferred through
subsequent enrichment. Universe artwork is owned by Universe + root QID;
fictional-entity artwork is owned by FictionalEntity + internal GUID.
Every root and entity route is constrained by an owned work carrying canonical
narrative_root_qid/fictional_universe_qid provenance and the normal
resource-authorization policy. A caller-supplied raw QID is not access proof.
Direct Event P585/P580/P582 and P4895 canonical facts are projected into the
in-universe Timeline separately from media release dates. Approved plugin-lore
sources remain supplemental and appear in Sources with their plugin and URL.
| Key | Default | Purpose |
|---|---|---|
fetch_temporal_qualifiers | true | Include P580/P582 in Data Extension requests |
batch_sparql_size | 50 | Max entities per batch Data Extension call |
lineage_depth | 2 | Maximum relationship traversal depth |
Subtitle Text Index
The Engine reads the Library’s subtitle text so that a time in a film or episode can later be tied to the characters and places named at that time. It is plain matching: no AI and no network.
- Sources. The index reads every stored, normalised subtitle track in
text_tracks(whichever provider or sidecar supplied it; no source is named in the code), and the text subtitle streams embedded in the video file (extracted with FFmpeg, one per language, skipping forced and picture-based streams and any language a stored track already covers). The probe also stores the file’s chapter times inmedia_asset_chapters. - Lines.
SubtitleLineParser(hand-written, WebVTT, SRT and ASS) turns a track into time-stamped plain lines insubtitle_lines. - Names.
SubtitleNameMatchermatches the labels of the Universe entities that belong to the file (entities linked to the file’s Work or to the Works above it, and every entity of the Universe those Works sit in) against the lines, whole words only and ignoring case. A character’s given name and surname count as aliases (four letters or more, capitalised in the label), and a name two entities share is not matched. Mentions of one entity less than 30 seconds apart form one window, stored insubtitle_name_tags(start, end, first and last line, mention count). - When.
SubtitleIndexHostedServiceruns the work in the background only: after a start-up delay, never while ingestion is working, ten items at a time. Its to-do list is durable (subtitle_index_tracks,subtitle_index_assets); a changed stored track, or a Universe graph that grew (theentity_signaturediffers), is picked up on the next pass. Tags are data only: no screen shows them yet.
Query Efficiency
Three layers prevent redundant Wikidata calls:
- Skip-if-enriched - entity level. If
enriched_at IS NOT NULL, no new Data Extension call is made. - Provider response cache - HTTP level. SHA-256 of request URL is checked in
provider_response_cachebefore any outbound call. ETag revalidation on cache expiry. - Universe-level deduplication - if a QID is already known in the graph, it is not re-fetched from Wikidata.