cpython/Parser
Petr Viktorin 6f1d448bc1
gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520)
* Add an InternalDocs file describing how interning should work and how to use it.

* Add internal functions to *explicitly* request what kind of interning is done:
  - `_PyUnicode_InternMortal`
  - `_PyUnicode_InternImmortal`
  - `_PyUnicode_InternStatic`

* Switch uses of `PyUnicode_InternInPlace` to those.

* Disallow using `_Py_SetImmortal` on strings directly.
  You should use `_PyUnicode_InternImmortal` instead:
  - Strings should be interned before immortalization, otherwise you're possibly
    interning a immortalizing copy.
  - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
    `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
    backports, as they are now part of public API and version-specific ABI.

* Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

* Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
  - `_Py_ID`
  - `_Py_STR` (including the empty string)
  - one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

* Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

* Intern `_Py_STR` singletons at startup.

* For free-threaded builds, intern `_Py_LATIN1_CHR` singletons at startup.

* Beef up the tests. Cover internal details (marked with `@cpython_only`).

* Add lots of assertions

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2024-06-21 17:19:31 +02:00
..
lexer gh-114569: Use PyMem_* APIs for most non-PyObject uses (#114574) 2024-01-26 10:11:35 +00:00
tokenizer GH-110749: fix unistd.h import in file_tokenizer.c (#110750) 2023-10-12 07:52:13 +02:00
action_helpers.c gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520) 2024-06-21 17:19:31 +02:00
asdl_c.py gh-118851: Default ctx arguments to AST constructors to Load() (#118854) 2024-05-09 15:30:14 -07:00
asdl.py bpo-43651: PEP 597: Fix EncodingWarning in some tests (GH-25142) 2021-04-02 12:53:46 +09:00
myreadline.c gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727) 2023-10-12 09:34:35 +02:00
parser.c gh-119724: Revert "bpo-45759: Better error messages for non-matching 'elif'/'else' statements (#29513)" (#119974) 2024-06-03 18:10:15 -07:00
peg_api.c gh-110805: Allow the repl to show source code and complete tracebacks (#110775) 2023-10-13 09:25:37 +00:00
pegen_errors.c gh-115823: Calculate correctly error locations when dealing with implicit encodings (#115824) 2024-02-26 12:57:09 +00:00
pegen.c gh-113993: Allow interned strings to be mortal, and fix related issues (GH-120520) 2024-06-21 17:19:31 +02:00
pegen.h gh-119118: Fix performance regression in tokenize module (#119615) 2024-05-28 19:17:49 +00:00
Python.asdl gh-116126: Implement PEP 696 (#116129) 2024-05-03 06:17:32 -07:00
string_parser.c gh-111380: Show SyntaxWarnings only once when parsing if invalid syntax is encouintered (#111381) 2023-10-27 12:19:34 +09:00
string_parser.h gh-102856: Initial implementation of PEP 701 (#102855) 2023-04-19 11:18:16 -05:00
token.c gh-107015: Remove async_hacks from the tokenizer (#107018) 2023-07-26 16:34:15 +01:00