feat(uploads): transparent presigned direct-to-storage upload#130
Conversation
| if progress is not None: | ||
| with lock: | ||
| done[0] = min(done[0] + length, total) | ||
| current = done[0] | ||
| progress(current, total) |
There was a problem hiding this comment.
nit: (not blocking) The progress snapshot is taken under lock, but progress(current, total) is invoked outside it. Two workers can compute current (e.g. 10 then 20) under the lock, release, and then race to call the callback — delivering (20) before (10). That contradicts the documented "monotonically non-decreasing" guarantee for bytes_done_total (see the UploadProgress docstring, lines 117–122). Moving the call inside the with lock: block makes delivery order match the computed order:
| if progress is not None: | |
| with lock: | |
| done[0] = min(done[0] + length, total) | |
| current = done[0] | |
| progress(current, total) | |
| if progress is not None: | |
| with lock: | |
| done[0] = min(done[0] + length, total) | |
| progress(done[0], total) |
The callback is already expected to be cheap (and the docstring tells callers to do their own locking), so serializing the calls is a fair trade for the ordering guarantee.
There was a problem hiding this comment.
Reviewed the direct-to-storage upload flow in hotdata/uploads.py and supporting changes. The prior progress-ordering nit is resolved (callback now fires inside the lock). Concurrency bounds, header isolation, short-read guards, and the retries-disabled finalize all look correct. LGTM.
Upload files directly to storage
Adds first-class file uploads to the Python SDK.
upload_filehandles the entire flow for you — it streams your data straight to object storage and returns the finalized result, with no extra round-trips through the API:Highlights
bytes, or any seekable binary file objectUploadErrorfor easy failure handlingupload_streamcovers the fallback for non-seekable streamsSee the README and CHANGELOG for full details.