Skip to content

Add hardware timestamp reporting#86

Draft
C-Achard wants to merge 12 commits into
cy/improve-encoding-speedfrom
cy/hardware-timestamps
Draft

Add hardware timestamp reporting#86
C-Achard wants to merge 12 commits into
cy/improve-encoding-speedfrom
cy/hardware-timestamps

Conversation

@C-Achard

@C-Achard C-Achard commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds frame capture metadata and recording timestamp support, with best-effort Basler hardware timestamp integration.

This PR refactors camera reads to allow returning extra objects alongside the frame, such as the existing host-side software timestamp, as well as optional backend-provided timestamp metadata.
The software timestamp remains the primary source, while hardware timestamps are added only when the backend can report them.

Main changes

  • Introduces a CapturedFrame wrapper for camera backends
  • Adds generic timestamp metadata support
    • Supports hardware camera-clock timestamps, raw device ticks, optional wall-clock timestamps, and backend-specific extras
  • Adds best-effort Basler hardware timestamp support
    • Reads device ticks count reported by Basler grab results when available
    • Falls back to an assumed one gigahertz tick frequency when the camera does not report one, and records that assumption in metadata. See Basler docs, generally devices which cannot report tick frequency operate at 1GHz, hence the fallback.
    • Treats zero-valued Basler timestamps as unavailable, which handles cases such as pylon emulation or cameras that expose the API but do not provide usable timestamps, as per Basler docs.
  • Updates non-Basler backends to return the new captured-frame object.
    • Aravis, GenTL, and OpenCV currently preserve software timestamps and do not provide hardware timestamp metadata for now.
  • Threads timestamp metadata through the recording path.
    • RecordingManager forwards timestamp metadata into VideoRecorder.
  • Updates recorder timestamp sidecar output.
    • Timestamp sidecars now use schema version 2.
    • Per-recording timestamp source metadata is stored once.
    • Hardware timestamp fields are omitted when unavailable.
  • Improves Basler exposure compatibility
    • Sets both ExposureTime and ExposureTimeAbs when writable as they both control the same property on different devices

What this does not do yet

  • Hardware timestamps are best effort and backend-dependent.
  • This does not change recording throughput or move recorder enqueueing off the GUI thread.

Testing

Added and updated tests accordingly; tested and confirmed to work as intended with 1x a2A1920-160umPRO.

C-Achard added 11 commits June 29, 2026 14:18
Introduce `FrameTimestampMetadata` in `dlclivegui/utils/timestamps.py` to standardize optional backend/hardware timestamp data for captured frames. The dataclass captures source/backend fields, converted and raw timestamp values, conversion metadata, and backend extras, and adds helper methods to serialize source-level data, per-frame values, full dictionaries, and the configured default reported timestamp.
Adds end-to-end support for optional per-frame hardware timestamp metadata. Camera backends now return a `CapturedFrame` object (while preserving tuple unpacking), multi-camera signals and recording paths carry timestamp metadata, and `VideoRecorder` persists richer timestamp records. The timestamp JSON output is upgraded to schema v2 with backward-compatible software `timestamps` plus source metadata and per-frame hardware timestamp fields.
Refactors backend tests to use the new `read()` return payload (`CapturedFrame`) instead of tuple unpacking, including frame/timestamp access updates and minor unused-variable cleanup. Test fixtures were also aligned with timestamp metadata support by returning `CapturedFrame` in the fake backend and extending fake recorder/frame callback signatures to accept `timestamp_metadata`.
Updates the recording metadata schema in `video_recorder.py` by renaming `hardware_frame_timestamps` to `frame_timestamps`. This aligns timestamp data with a more general key name while preserving the same underlying values.
Expands test coverage for frame timestamp metadata end-to-end: Basler backend reads now validate hardware timestamp extraction, controller/recording manager tests assert metadata forwarding, and video recorder tests verify schema_version 2 sidecar output for both software-only and hardware-backed timestamps. Also adds focused unit tests for `FrameTimestampMetadata` source/frame field splitting and default-reported value behavior.
Improve Basler hardware timestamp robustness by treating support as best-effort, recording the tick-frequency source, falling back to an assumed 1 GHz clock when frequency is unavailable, and ignoring zero-value camera timestamps as missing data. The frame timestamp metadata now includes the frequency source in `extra`, and unused last-frame timestamp state was removed. Video recorder metadata also drops the legacy top-level `timestamps` field in favor of the structured `timestamp_sources` schema.
Updates `test_video_recorder.py` to stop asserting the top-level `timestamps` list in sidecar JSON fixtures. The tests now focus on schema v2 fields that remain authoritative (`frame_timestamps`, `start_time`, `end_time`, `duration_seconds`, and timestamp source metadata), aligning expectations with current sidecar output.
@C-Achard C-Achard requested a review from Copilot June 29, 2026 14:10
@C-Achard C-Achard self-assigned this Jun 29, 2026
@C-Achard C-Achard added enhancement New feature or request camera Related to cameras and camera backends labels Jun 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds optional per-frame timestamp metadata (including best-effort Basler hardware timestamps) and threads that metadata through capture, controller signals, recording, and the timestamp sidecar output format.

Changes:

  • Introduces CapturedFrame as the standard return type for camera backend read() calls, carrying software timestamp plus optional timestamp metadata.
  • Adds FrameTimestampMetadata and integrates Basler tick-based timestamps (with tick-frequency discovery/fallback) into capture and recording.
  • Updates VideoRecorder timestamp sidecar JSON to schema v2 with per-recording timestamp source metadata and per-frame timestamp records.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/utils/test_timestamps.py Adds unit tests for FrameTimestampMetadata dict splitting and default selection behavior.
tests/services/test_video_recorder.py Updates/extends tests for schema v2 timestamp sidecar output and hardware metadata inclusion.
tests/services/test_multicam_controller.py Adjusts signal handler signatures and adds coverage for forwarding timestamp metadata.
tests/gui/test_rec_manager.py Adds coverage ensuring RecordingManager.write_frame() forwards timestamp metadata to recorder.
tests/conftest.py Updates test backend and recorder fakes to use CapturedFrame and pass through timestamp metadata.
tests/cameras/backends/test_opencv_backend.py Updates OpenCV backend tests for CapturedFrame return type.
tests/cameras/backends/test_gentl_trigger.py Updates GenTL trigger test to read .frame from CapturedFrame.
tests/cameras/backends/test_gentl_backend.py Updates GenTL backend tests for CapturedFrame return type.
tests/cameras/backends/test_basler_backend.py Updates Basler backend tests and adds coverage for Basler timestamp metadata.
tests/cameras/backends/test_aravis_backend.py Updates Aravis backend tests for CapturedFrame return type and minor unused var cleanup.
tests/cameras/backends/conftest.py Extends fake Basler SDK objects to provide timestamp and tick-frequency nodes.
dlclivegui/utils/timestamps.py Introduces FrameTimestampMetadata dataclass and serialization helpers.
dlclivegui/services/video_recorder.py Threads timestamp metadata through queue/write loop and writes schema v2 timestamp sidecar JSON.
dlclivegui/services/multi_camera_controller.py Extends capture/recording signals and internal plumbing to propagate timestamp metadata.
dlclivegui/gui/recording_manager.py Forwards timestamp metadata into VideoRecorder.write().
dlclivegui/gui/main_window.py Passes timestamp metadata from controller signal into recording manager.
dlclivegui/cameras/base.py Adds CapturedFrame, updates backend interface to return it, and extends capabilities.
dlclivegui/cameras/backends/opencv_backend.py Updates OpenCV backend to return CapturedFrame on success/failure paths.
dlclivegui/cameras/backends/gentl_backend.py Updates GenTL backend to return CapturedFrame.
dlclivegui/cameras/backends/basler_backend.py Adds Basler timestamp tick-frequency handling and per-frame timestamp metadata generation.
dlclivegui/cameras/backends/aravis_backend.py Updates Aravis backend to return CapturedFrame.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dlclivegui/services/video_recorder.py Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

camera Related to cameras and camera backends enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants