Start improving high-throughput multi-camera recording path and diagnostics#85
Draft
C-Achard wants to merge 11 commits into
Draft
Start improving high-throughput multi-camera recording path and diagnostics#85C-Achard wants to merge 11 commits into
C-Achard wants to merge 11 commits into
Conversation
Switch Basler camera startup to `pylon.GrabStrategy_OneByOne` instead of `LatestImageOnly`, and update the nearby identity-persistence comment for clarity.
Add ignore patterns for generated profiling files (`profile*.svg`, `scalene*.json`, and `scalene*.html`) so local performance analysis outputs are not accidentally committed.
Introduce a new `profiling` optional dependency group in `pyproject.toml` and include `scalene` so profiling tools can be installed independently from test and framework extras.
Adds a dedicated full-rate `recording_frame_ready` signal path so recording is decoupled from the inference/display frame flow, reducing processing overhead during capture. The GUI now exposes a fast-encoding toggle and persists it into recording settings, and recorder startup passes codec-specific writer options through to `VideoRecorder`. Recording telemetry was expanded to report enqueued vs written frames, writer FPS, queue fill against buffer size, backlog, and drops, improving visibility into recording throughput and pressure.
Turns on timing logging for multi-camera worker, recorder, and Basler backend diagnostics. Recording settings now include a `fast_encoding` flag, with `writegear_options` made more robust for missing/invalid FPS and optional low-latency FFmpeg options (`ultrafast` + `zerolatency`) for x264/x265. Recorder stats were expanded with buffer capacity awareness (`buffer_size`), derived backlog/fill-ratio properties, and richer formatted output showing queue fill and backlog.
Updates test fixtures and unit tests around recording flow changes: FakeVideoRecorder now mirrors new constructor/runtime fields, Basler fake includes one-by-one grab strategy, and GUI/controller tests validate recording-frame emission gating plus overlay recording via `_on_recording_frame_ready`. Tests also cover `RecordingSettings.writegear_options` (including fast x264 options and FPS fallback), RecordingManager writer option wiring, and richer recorder stats formatting/aggregation with backlog and queue capacity output.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves multi-camera recording robustness and diagnostics under high-throughput load by adding a dedicated full-rate recording signal/path, enhancing recorder backlog/queue reporting, and introducing optional “fast encoding” FFmpeg parameters via centralized WriteGear option generation.
Changes:
- Added a dedicated
recording_frame_ready(camera_id, frame, timestamp)signal and routed recording through a leaner handler in the main window. - Expanded recorder stats to include buffer capacity, queue fill, and backlog; updated aggregated multi-camera summaries accordingly.
- Added
RecordingSettings.writegear_options()(incl. optional “fast encoding”), updated Basler grab strategy, and extended tests/ignores/extras to support these features.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/utils/test_stats.py | Updates recorder stats formatting tests for backlog/buffer-capacity output. |
| tests/test_config.py | Adds unit tests for RecordingSettings.writegear_options() and fast-encoding behavior. |
| tests/services/test_multicam_controller.py | Adds test for recording-frame emission enable/disable behavior. |
| tests/gui/test_rec_manager.py | Updates multi-recorder summary aggregation tests and checks writer options propagation. |
| tests/gui/test_pose_overlay.py | Updates overlay-recording test to use the new recording-frame handler. |
| tests/conftest.py | Extends FakeVideoRecorder to accept new constructor args (buffer/writer options). |
| tests/cameras/backends/conftest.py | Updates Basler fake to include GrabStrategy_OneByOne. |
| pyproject.toml | Adds optional “profiling” dependency group. |
| dlclivegui/utils/stats.py | Adds recorder backlog/fill helpers and extends recorder stats formatting output. |
| dlclivegui/services/video_recorder.py | Adds writer_options, improves logging, and adjusts stats computation. |
| dlclivegui/services/multi_camera_controller.py | Adds recording_frame_ready signal and togglable emission during capture. |
| dlclivegui/gui/recording_manager.py | Centralizes writer option generation and expands stats aggregation/summaries. |
| dlclivegui/gui/main_window.py | Adds “fast encoding” UI toggle, connects new recording signal, and routes recording via lean handler. |
| dlclivegui/config.py | Adds fast_encoding to recording settings; changes timing-log defaults. |
| dlclivegui/cameras/backends/basler_backend.py | Switches Basler grab strategy and gates trigger debug logging behind a flag. |
| .gitignore | Ignores common profiling artifacts (scalene/profile outputs). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Wire the fast encoding checkbox to QSettings so its value is restored on startup and saved when toggled. This adds typed get/set helpers for `recording/fast_encoding` in `DLCLiveGUISettingsStore`, updates `main_window` to prefer persisted values over config defaults, and includes a roundtrip unit test for the new setting. It also removes an obsolete commented-out recording block in the frame processing path.
Remove defensive `hasattr` checks when connecting recording settings signals in `DLCLiveMainWindow`. The widgets are expected to exist, so connecting unconditionally avoids silently skipping persistence and recording path preview updates if an expected widget is missing or renamed.
Turns off multi-camera, recorder, and Basler timing logs by default to reduce debug noise in normal runs. Also cleans up stale priority wording in main window comments, updates VideoRecorder docstrings to reflect `writer_options`, and fixes stats tests to import `RecorderStats` from `dlclivegui.utils.stats`.
Ensure ffmpeg writer defaults (`-input_framerate`, `-vcodec`, `-crf`) are always applied, even when custom writer options are provided, while still allowing overrides via `writer_options`. Also improve recorder stats by estimating `buffer_seconds` from average or last frame latency when write FPS is unavailable, avoiding zero/underreported buffer duration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improves multi-camera recording behavior under high-throughput conditions by separating recording frame delivery from the heavier GUI/DLC processing path, adding clearer recorder backlog/queue diagnostics, and introducing an optional faster encoding mode for supported FFmpeg codecs.
This is a first quick remediation to the deeper architectural concern requiring the recording to be moved closer to camera SDK frame grabbing.
Main changes
For libx264/libx265, this applies:
What this improves
This makes it easier to diagnose when the recorder is falling behind before drops occur. Previously, dropped = 0 could hide the fact that recorder queues were filling. The new stats expose that state directly through queue fill and backlog.
The fast encoding option also provides a user-controlled way to trade compression efficiency/file size for higher recording throughput.
Not solved yet
Testing
Added/updated unit tests for: