random/hashlib errors on AMD64 RHEL8 FIPS No Builtin Hashes 3.x

[vstinner]

The following code in Lib/random.py causes some warnings and test failures:

try:
    # hashlib is pretty heavy to load, try lean internal module first
    from _sha2 import sha512 as _sha512
except ImportError:
    # fallback to official implementation
    from hashlib import sha512 as _sha512

Many tests fail on AMD64 RHEL8 FIPS No Builtin Hashes 3.x:

16 tests failed:
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_shutdown
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_spawn.test_misc
    test.test_multiprocessing_spawn.test_processes test___all__
    test_compileall test_hashlib test_logging
    test_multiprocessing_main_handling test_regrtest test_tempfile
    test_threading test_venv

build: https://buildbot.python.org/all/#/builders/480/builds/6174

Example of test failure:

FAIL: test_del_on_shutdown_ignore_errors (test.test_tempfile.TestTemporaryDirectory.test_del_on_shutdown_ignore_errors)
Test ignoring errors works when a tempdir is gc'ed on shutdown.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/test/test_tempfile.py", line 1765, in test_del_on_shutdown_ignore_errors
    self.assertNotIn("Error", err)
AssertionError: 'Error' unexpectedly found in 'ERROR:root:code for hash blake2b was not found.\n
Traceback (most recent call last):\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/random.py", line 68, in <module>\n
    from _sha2 import sha512 as _sha512\n
ModuleNotFoundError: No module named \'_sha2\'\n
\n
During handling of the above exception, another exception occurred:\n
\n
Traceback (most recent call last):\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 245, in <module>\n
    globals()[__func_name] = __get_hash(__func_name)\n
                             ^^^^^^^^^^^^^^^^^^^^^^^\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 129, in __get_openssl_constructor\n
    return __get_builtin_constructor(name)\n
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 123, in __get_builtin_constructor\n
    raise ValueError(\'unsupported hash type \' + name)\n
ValueError: unsupported hash type blake2b\n
ERROR:root:code for hash blake2s was not found.\n
Traceback (most recent call last):\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/random.py", line 68, in <module>\n
    from _sha2 import sha512 as _sha512\n
ModuleNotFoundError: No module named \'_sha2\'\n
\n
During handling of the above exception, another exception occurred:\n
\n
Traceback (most recent call last):\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 245, in <module>\n
    globals()[__func_name] = __get_hash(__func_name)\n
                             ^^^^^^^^^^^^^^^^^^^^^^^\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 129, in __get_openssl_constructor\n
    return __get_builtin_constructor(name)\n
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 123, in __get_builtin_constructor\n
    raise ValueError(\'unsupported hash type \' + name)\n
ValueError: unsupported hash type blake2s\n
/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/tempfile.py:909: ResourceWarning: Implicitly cleaning up <TemporaryDirectory \'/tmp/test_python_zoi2ne02/tmpzl8m50ty/tmp9_o6qzyc\'>\n
  _warnings.warn(warn_message, ResourceWarning)\n
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=\'/tmp/test_python_zoi2ne02/tmpzl8m50ty/tmp9_o6qzyc/test_dir/open_file.txt\' mode=\'w\' encoding=\'UTF-8\'>\n
'

There are many logs like:

ERROR:root:code for hash blake2b was not found.
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/random.py", line 68, in <module>
    from _sha2 import sha512 as _sha512
ModuleNotFoundError: No module named '_sha2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 245, in <module>
    globals()[__func_name] = __get_hash(__func_name)
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 129, in __get_openssl_constructor
    return __get_builtin_constructor(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type blake2b
ERROR:root:code for hash blake2s was not found.
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/random.py", line 68, in <module>
    from _sha2 import sha512 as _sha512
ModuleNotFoundError: No module named '_sha2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 245, in <module>
    globals()[__func_name] = __get_hash(__func_name)
            
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 129, in __get_openssl_constructor
    return __get_builtin_constructor(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-fips-x86_64.no-builtin-hashes/build/Lib/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type blake2s

Linked PRs

• gh-110357: hashlib no longer logs at import when a guaranteed hash is unavailable #152538
• [3.15] gh-110357: hashlib no longer logs at import when a guaranteed hash is unavailable (GH-152538) #152554

[vstinner]

IMO a first simple fix would be to avoid always importing hashlib on import random, since it's only needed when random.seed() is called with a string.

[vstinner]

Sadly, I don't have time to dig into this issue. I prefer to close it.