How File Ingestion Works

Learn what happens after you add a file to Tuvima Library. This three-minute overview explains why a file may appear in Operations before you can browse it.

Follow a new file

The Engine waits for copying to finish, fingerprints the file, and reads its embedded details. It resolves ambiguous types, such as a song versus an audiobook track, before matching.

Provider matching gathers identity clues and artwork. Wikidata can then add structured identity and relationships. An uncertain match may need review rather than an automatic guess.

Understand when it becomes visible

Browsing needs a real title, a resolved media type, and a settled artwork result. Settled can mean a cover is present or an artwork pass has confirmed that none is available.

Visibility and file organization are separate milestones. Existing/read-only sources remain in place; permitted managed sources follow their organization rules. Personal View media uses a separate local workflow.

Check progress or act

Open Operations from the header activity indicator to follow active work. The screen refreshes automatically; Check folders for changes (in the Ingestion page ”⋯” menu) starts an extra folder scan. Use Review Queue only when the item needs your decision.

A completed file count does not mean the whole run has finished. Required identity, enrichment, or organization operations can still be active.

Technical details

When a new file lands in a watched folder, Tuvima Library does not trust it immediately. The Engine waits until the file is stable, reads what it can from the file itself, applies source-policy-aware staging or in-place indexing, and then starts deciding whether it is ready for the Dashboard and the organized library.

The big picture

File appears in a watched folder
  -> settle and lock check
  -> fingerprint
  -> scan embedded metadata
  -> classify ambiguous formats
  -> source-policy-aware staging or indexing
  -> Stage 1 Retail
  -> Stage 2 Wikidata
  -> Quick Hydration
  -> Stage 3 enrichment
  -> managed artwork settlement
  -> browse readiness gate
  -> library organisation

The key idea is simple: trust must be earned.

Step 1: Detection and settling

The Engine notices a new file quickly, but it does not scan it immediately. Large files can take time to finish copying. The system waits for the file to stop changing and confirms it is no longer locked by another process.

That avoids half-read files and noisy filesystem events.

Step 2: Fingerprinting

The Engine computes a SHA-256 hash from the file contents.

That fingerprint lets the system:

recognise the same file even if it is renamed or moved
detect duplicates
attach later rescans to the same underlying asset

The fingerprint is the file’s durable identity, not its filename.

Step 3: Scanning

The correct processor opens the file and extracts embedded metadata such as:

title
author or creator
year
narrator
series information
embedded cover art
file-format-specific tags

These values become metadata claims with confidence scores. A clean EPUB title is more trustworthy than a guessed filename title, and that difference matters later.

Step 4: Classifying ambiguous formats

Some formats are ambiguous. An MP3 could be a song or an audiobook chapter. An MP4 could be a movie or a TV episode.

The Engine combines file signals, folder hints, and AI-assisted classification to resolve those cases. If it still cannot make a safe call, the item goes to review instead of being forced into the wrong media type.

Step 5: Safe staging

Managed intake can stage files before final organization. Existing/read-only sources are indexed in place and are not moved into staging.

Staging is the Engine’s safe holding area:

the original file is preserved while matching and enrichment run
retries can happen without touching the organised library
a bad or uncertain match does not immediately rewrite your final folder structure

This staging decision is separate from whether the item is visible in the main browse surfaces.

Step 6: Identity matching

After scan and staging, the Engine starts the two identity stages:

Stage 1 Retail finds practical provider candidates, artwork, people, ratings, descriptions, and bridge IDs
Stage 2 Wikidata resolves canonical identity from those bridge IDs

Retail matching uses these confidence gates:

>= 0.90 can be auto-accepted
0.65 to < 0.90 goes to review
< 0.65 is treated as too weak

That stricter gate reduces false positives and improves the quality of later Wikidata resolution.

If Retail cannot produce a safe match, Wikidata is not used as a broad text fallback. The item goes to review instead. If Retail succeeds but no QID is found, the item keeps its retail data and can be retried later. Missing QID alone need not create a review item.

Step 7: Artwork settlement

Quick Hydration gets the item visible with core identity and accepted artwork when enough facts are available. Managed artwork is stored under .data/assets/... and referenced from the database; media-folder sidecars are optional exports only.

The Engine also tracks whether artwork is:

present
still pending
explicitly missing

That matters because the main browse surfaces do not show an item until its artwork outcome has settled. A file can be safely staged and partly identified without being ready for the main browse surfaces yet.

Stage 3 enrichment continues in the background after the fast path. It expands people, fictional entities, narrative roots, relationships, lyrics/subtitles, extra artwork, and other universe details.

Step 8: Browse visibility

An item becomes visible in the main browse surfaces only after it passes the browse readiness gate:

non-placeholder title
resolved media type
settled artwork outcome

If it fails that gate, it stays visible in Operations and, when a decision is actionable, Review Queue until the missing piece is resolved.

This is why “the system has seen the file” and “the file is in the main browse surfaces” are no longer the same moment.

On the live ingestion page a row that has not passed the gate says what it is still waiting for, in this order: matching, its show or album (TV episodes and music tracks only), and artwork. The page uses the same Ready rule as the browse surfaces, so a row is Ready exactly when the item can appear there.

Step 9: Organisation

Promotion into the organised library on disk is a later decision based on the wider confidence and organisation rules.

So there are two separate milestones:

the item is ready enough to surface in the main browse surfaces
the item is ready enough to be promoted into the final organised library structure

That separation makes the product more honest and reduces bad auto-moves.

The Import folder

You can set one optional Import folder where new media is dropped, for example by Sonarr or Radarr. Tuvima creates a folder inside it for each kind of media (Movies, TV, Music, Books, Audiobooks and Comics), and a file in one of them is treated as that kind. Files arrive, settle and are identified like any other, then move into the one managed folder you marked New media for their kind, named the usual way.

If that folder is offline, full or can’t be written to, or the kind has no managed folder yet, the file simply stays where it is, untouched, and is filed on a later pass once the folder is back. Files Tuvima can’t match yet stay in the Import folder and appear in Review with a “From the Import folder” badge, and are filed once you resolve them. An exact duplicate follows your duplicate rule: with Keep one copy only the copy in the Import folder is removed, and with Keep both it is filed next to the other with a different name. Nothing in your library folders is ever deleted by this.

When things go wrong

The pipeline is designed to fail safely:

if the file never settles, it is not processed prematurely
if the processor cannot read it cleanly, the system falls back to weaker signals
if Retail cannot find a safe match, the item goes to review

Watching Operations

Settings > Server > Library and Ingestion is the single operational surface at /settings/ingestion. It opens on the library summary (counts by type and health), then the live ingestion panel, then Now (the few items being worked on at this moment) and Added (what was fully added on each of the last few days, by type). Every Added count and its “View all” link open the Activity log on its Added filter. It answers what is happening now, whether the pipeline is waiting, whether anything needs attention, and what was added recently. Needs Review remains the human decision queue, on its own page at /settings/review that the administrator banner opens.

It shows real application state from the Engine:

the groups being added now, and what became Ready on each recent day
registered, provisional, and review lifecycle counts
current outcomes: added, needs review and failed
Watch, Listen, and Read source folders from config/libraries.json
provider health without exposing secrets
pipeline counts from durable identity jobs and ingestion logs
grouped review reasons from pending Review Queue records

File progress and run completion are deliberately separate. The Files checked outcome reports intake volume, while the prominent overall bar combines measurable pipeline stages for the logical run. It stays below completion while a stage is active and shows the current stage’s own task count beneath it. The run stays active while required identity, artwork, people, relationship, or organization operations remain outstanding.

The page’s headline, file tally, progress bar and time estimate are the Engine’s IngestionQueueProgress snapshot, pushed over SignalR whenever its numbers change and read again on every (re)connect; the Dashboard shows it as sent and works out no percentage or estimate of its own. The cards below come from the Engine’s presentation view, which the Dashboard refreshes more often while work is active and less often when idle. If the Engine stops answering or the live connection drops, the page says “The Library’s engine isn’t responding” with a Retry button and greys the last numbers until a fresh snapshot arrives. If a signal is not tracked yet, the page says so instead of inventing a count. The top navigation activity indicator opens Ingestion for authorized system work. The page has no manual status-refresh control because this synchronization is automatic. Its one Check folders for changes (in the Ingestion page ”⋯” menu) action starts an extra scan of watched folders; folder monitoring, schedules, and queued processing continue automatically.

Recently Added loads the newest items first. Search and quick filters query the server, while Show older appends another bounded page. A show, album or series is one row however many times its files were found; opening a row shows its tracks, episodes or parts without loading raw technical records.

On a phone, Ingestion keeps current state, active and queued operation counts, attention, and compact rows visible. Needs Review resolution is reserved for desktop and tablet, while its unresolved count remains visible beside Operations. All responsive summaries use the same state and counts as desktop.

if Wikidata finds no QID, the item can still remain usable without forcing a bad identity
if artwork is still unresolved, the item stays out of the main browse surfaces until that question is settled