Reference-count ceiling is 32 767 on 32-bit targets: refcount_overflow aborts the process #8469

On any 32-bit target — wasm32-wasip2 here — an object reaching 32 767 live references calls refcount_overflow(), which is std::process::abort(). Under wasm that is an unreachable trap, so the embedder gets no Python-level error and no message.

Reproducer

marker = object()
holder = [marker] * 32767

32767 aborts. 32760 returns normally. Bisected precisely: [marker] * 32765 returns, * 32766 traps — 32 767 total live references at the trap, counting the ones the interpreter holds itself.

Reproduced with RustPython 0.5.0 embedded in a wasm32-wasip2 component (rustpython-vm with compiler, freeze-stdlib, host_env, stdio), run under wasmtime 45, built by Rust 1.96.1 stable.

Symbolised guest backtrace, from a release build with -C strip=none:

0: abort
1: std::sys::pal::wasi::abort_internal
2: std::process::abort
3: rustpython_common::refcount::refcount_overflow
4: rustpython_vm::object::payload::PyPayload::into_ref
5: rustpython_vm::vm::context::Context::new_str
6: rustpython_vm::frame::ExecutingFrame::execute_make_function

Cause

crates/common/src/refcount.rs packs the reference count and its flags into one usize, so the counter widths follow the pointer width:

// State layout (usize):
//   [1 bit: destructed] [1 bit: published] [1 bit: leaked] [N bits: weak_count] [M bits: strong_count]
// 64-bit: N=30, M=31.  32-bit: N=14, M=15.

15 bits gives 32 767 strong references at 32-bit pointer width, against roughly 2.1 billion at 64-bit. The weak count is narrower still: 14 bits, 16 383.

main carries the same layout as the 0.5.0 tag.

Why this is reachable in ordinary use, not a synthetic limit

Interned strings and shared singletons cross 32 767 once a realistic module graph is loaded. Measured inside the component:

• import urllib3 alone succeeds, and performs a real HTTP request.
• import urllib3 followed by import decimal aborts. So does import http.cookiejar, and so does import charset_normalizer, in either order. Each of those imports alone succeeds.
• import requests therefore aborts, which puts most of the third-party ecosystem out of reach on 32-bit targets.

Three other explanations were tested and refuted, so the ceiling is not standing in for something else:

• Stack. Rebuilding with -C link-arg=-zstack-size=16777216, verified applied — the __stack_pointer global moves from 1048576 to 16777216 — changes nothing.
• Recursion limit. sys.setrecursionlimit at 256, 400 and 600 abort identically. A debug build, whose limit is 256, raises RecursionError on the same script only because it stops before the refcounts climb.
• Memory. A 512 MiB allocation succeeds after import urllib3.

What a fix has to preserve

The compare-and-swap runs over the whole packed word, which also carries the weak count, DESTRUCTED, LEAKED and PUBLISHED. Widening the strong count on 32-bit therefore cannot be a field-width tweak alone. Two directions, both with costs worth weighing by someone who knows the GC design:

1. Use a 64-bit state word on 32-bit targets, paying for a 64-bit atomic where the platform lacks one natively.
2. Split the flags out of the counter word, so both counts get the full width.

Related for context rather than as a duplicate: rust-lang/rust#108706 makes the same observation about Arc — an overflow that is theoretical at 64-bit is reachable at 32-bit.

Downstream tracking issue, with the full measurement log: https://gitlab.com/lx-industries/openblob/-/issues/590