gh-152718: Reject oversized table counts in the profiling binary reader

[tonghuaroot]

This bounds the eager string-table and frame-table allocations in the
_remote_debugging binary profile reader against the file size before
allocating, so a .pyb file whose footer declares an oversized count is
rejected with a ValueError instead of triggering a multi-gigabyte allocation.

Each table entry occupies at least one byte on disk (a string is a >=1-byte
length varint, a frame is six >=1-byte varints plus the opcode byte), so a count
larger than (file_size - table_offset) / MIN_ENTRY_SIZE cannot be backed by
real data. This mirrors the existing RLE-count bound in binary_reader_replay.
Legitimate files are unaffected.

Regression tests in test_binary_format patch the footer counts and assert the
reader rejects both an oversized and a one-past-capacity count, while a count
exactly at the cap still opens.

• Issue: _remote_debugging: unbounded allocation when a profiling binary file declares oversized table counts #152718

[pablogsal]

Thanks!

[maurycy]

Wondering about a helper. With these two and this one:

cpython/Modules/_remote_debugging/binary_io_reader.c

Lines 1056 to 1066 in ecdef17

	/* Validate RLE count to prevent DoS from malicious files.
	* Each RLE sample needs at least 2 bytes (1 byte min varint + 1 status byte).
	* Also reject absurdly large counts that would exhaust memory. */
	size_t remaining_data = reader->sample_data_size - offset;
	size_t max_possible_samples = remaining_data / 2;
	if (count > max_possible_samples) {
	PyErr_Format(PyExc_ValueError,
	"Invalid RLE count %u exceeds maximum possible %zu for remaining data",
	count, max_possible_samples);
	return -1;
	}

it'd be three. Something along the lines of reader_validate_count("RLE", count, reader->sample_data_size - offset, 2) etc.

Generally, wondering about centralizing all these validations in a single place.

[tonghuaroot]

Good idea — done in 572947c: the string, frame and RLE checks now share a reader_validate_count(what, count, available, min_entry_size) helper. One heads-up: the RLE call sits in the region #152722 also touches, so whichever lands first the other will need a small rebase there. Thanks!

[pablogsal]

@tonghuaroot seems tests are failing can you take a look?

[tonghuaroot]

Thanks @pablogsal, pushed a fix in 7c23d37. The failure was 32-bit only (Win32 and Emscripten): test_open_rejects_frame_count_larger_than_file patches the frame count to 0xFFFFFFFF, which on a 32-bit build overflows frames_count * sizeof(FrameEntry), so the pre-existing SIZEOF_SIZE_T < 8 guard raises OverflowError before the new count-vs-file-size check raises ValueError. Both reject the file, so I relaxed the test to accept either. Re-running CI now.

[miss-islington-app]

Thanks @tonghuaroot for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

[pablogsal]

Thanks for te PR @tonghuaroot

[bedevere-app]

GH-153050 is a backport of this pull request to the 3.15 branch.