Vessel Identity · Model evaluation

See where the model is right, uncertain or confused.

The batch verifier runs an active vessel-identity profile against available validation and test recordings, preserves the expected MMSI and exposes the best alternative—not just a single headline accuracy number.

Analyze · Vessel Identity model
Analyze Vessel Identity library with Verify Model, Model History and Train or Retrain Model controls
Verification uses the selected named model profile and the library’s held-out recordings.

Verification workflow

Test the model against recordings it did not train on.

  1. Select a named vessel identity model profile.
  2. Open Verify Model from the Vessel Identity tab.
  3. Review the number of locally available validation and test files.
  4. Choose the unknown-vessel similarity cutoff.
  5. Run verification and inspect individual, encounter and confusion results.

Leakage protection: if an encounter contains any training recording, all validation or test clips from that encounter are excluded automatically.

Per-file evidence

A result table designed for investigation.

Expected MMSI

The library label remains visible beside every prediction so errors can be inspected directly.

Predicted identity

The strongest model match is reported as an MMSI, or as Unknown vessel when it falls below the cutoff.

Cosine similarity

The score measures proximity to the learned vessel prototype in the model’s acoustic embedding.

Strongest alternative

The runner-up MMSI and score reveal close competitors that a simple correct/incorrect label would hide.

Split and encounter

Validation or test origin and encounter ID remain visible for every evaluated recording.

Error visibility

Unreadable or incompatible files are marked as errors without concealing the rest of the batch.

Summary metrics

Read accuracy at two levels.

OutputQuestion answeredHow to use it
File accuracyHow many individual clips received the expected MMSI?Useful for reviewing difficult windows, but adjacent clips are correlated.
Encounter-majority accuracyHow many independent passages were identified by the majority of their clips?Use as the more conservative operational summary.
Unknown resultsHow often was no candidate similar enough?Tune the cutoff while balancing false identities against missed known vessels.
Confusion countsWhich expected vessels are repeatedly mapped to which alternatives?Find acoustically similar identities and under-represented training classes.
Alternative scoreWas the winner clear or nearly tied?Prioritise close decisions for listening and signature comparison.

Unknown-vessel rejection

A model should be allowed to say “unknown.”

Closed-set accuracy assumes every recording belongs to a vessel represented during training. Real deployments contain untrained vessels, overlapping traffic and non-vessel sound. Analyze therefore applies an adjustable similarity cutoff.

Higher cutoff

Requires a closer match and reduces unsupported identity assignments, but may reject more recordings from known vessels.

Lower cutoff

Returns a known candidate more often, but increases the chance of confidently assigning unfamiliar audio to the wrong identity.

The cutoff should be selected using independent known-vessel and out-of-library recordings representative of the deployment environment.

Good evaluation practice

What a convincing result requires.

Are many clips from one passage enough?

No. They provide useful acoustic variation within an encounter but do not replace independent passages recorded at different times, ranges and operating states.

Does a high similarity score prove identity?

No. It indicates similarity to a learned prototype. Confounding vessels, hydrophone response and environmental noise still require review.

Why report the alternative vessel?

Repeated near-ties expose pairs that need more training encounters, better separation or a coarser class such as vessel type.

Can results be compared across model versions?

Yes. Named profiles and model history allow the same held-out material to be rerun after retraining or restoration.

Verify vessel models against independent evidence.

We can walk through thresholds, alternatives, confusion patterns and encounter-level results.

Plan a demonstration