Add new type of executable to support external JITs #142598
Description
Feature or enhancement
Proposal:
#105727 introduced the ability to have different types of executables. During the discussion of that there was mention of potentially adding support for JITs to be able to plug in as well. There's a similar disucssion of the idea over here: faster-cpython/ideas#575
This extends the existing set of executors so that PEP-523 external tools can hook into the interpreter frames and create their own external frames which live on the stack with some minimal amount of initialization. This can be used by JIT compilers, tools that compile Python code to C (e.g. Cython, MyPyC), or other tools that would like to display frames in the Python stack.
The initialization that is required is minimal. There is a new _PyInterpreterFrameCore struct that is added which is the first field of the _PyInterpreterFrame struct - this cleanly captures the data which needs initialization and ensures the runtime isn't inadvertently accessing fields of uninitialized frames. The fields of the core structure are: f_executable, previous, and owner. While the frame only has the minimal values initialized the full size of the frame does need to be allocated so that reification doesn't fail - and the reifier is explicitly disallowed from returning a different pointer.
The f_executable is initialized to a value which is created by PyUnstable_MakeExternalExecutable This is created with a callback that is used to reify the full frame, a code object, and an optional piece of state.
The reifier is called in various places where the JIT needs to populate the frame - for example accessing globals, builtins, instr_ptr or creating a user-visible frame object. Currently the API just provides a single callback with no information on what's required. Future iterations could modify this to allow finer grained control if it was desired - but reification of the frame is generally going to happen in the slow path so simplicity seems the best here.
The code object is provided both for external introspection tools and for the runtime to provide the code object in reified frames. In the external introspection tools case reifier can not be invoked so the code object provides some minimal amount of information about the frame. Because the reifier may want callbacks to update the the frame state over time f_executable cannot be restored to the code object on first reification. Therefore the code object here also allows the runtime to properly create a user-visible Python frame object for the external frame complete with f_code.
The state object is provided for the reifier to provide any state which is necessary to reify the frame assuming it has no where else to store it. For example this could include the function object so that the reifier can populate functions, globals and builtins with simple reads. Another example is it could also include data which maps from IP address to instruction pointer if the reifier wanted to update instr_ptr on each callback. The reifier needs to store sufficient data here or somewhere else so that it can reify the frame without invoking other arbitrary Python code that could cause infinite recursion and can do so without failing.
The runtime will always call the reifier on an external frame before accessing members of that frame. This is captured in the reifier callback API where the reifier is transforming the _PyInterpreterFrameCore into a fully initialized _PyInterpreterFrame that respects all of the guarantees of a standard _PyInterpreterFrame. While this returns a pointer of a different type it is an error to return a pointer to a different address. The JIT doesn't need to continue to maintain the initialized state of the frame and the runtime will always call the reifier before further accesses at a different point in time. The reification must also not fail - the external frame owner must have all of the data required to initialize the frame. The reifier can zero initialize the locals, stackpointer, frame_obj.
The _frameowner enum is updated to note that a frame is externally owned. Technically this can be derived by the f_executable type but it is simpler to check the owner field. A flag is added to indicate that the frame is externally owned and can be applied to either frames owned by threads or generators. Frames owned by the frame object are never externally owned are neither are frames owned by the interpreter.
During GC external frames will not have their fields introspected and the reifier callback will not be invoked.
The runtime will consider the external frame to be "complete" - it will be visible in stack traces. This is because these frames are designed to be user-visible.
enum _frameowner {
// The frame is allocated on per-thread memory that will be freed or transferred when
// the frame unwinds.
FRAME_OWNED_BY_THREAD = 0x00,
// The frame is allocated in a generator and may out-live the execution.
FRAME_OWNED_BY_GENERATOR = 0x01,
// A flag which indicates the frame is owned externally. May be combined with
// FRAME_OWNED_BY_THREAD or FRAME_OWNED_BY_GENERATOR. The frame may only have
// _PyInterpreterFrameFields. To access other fields and ensure they are up to
// date _PyFrame_EnsureFrameFullyInitialized must be called first.
FRAME_OWNED_EXTERNALLY = 0x02,
// The frame is owned by the frame object (indicating the frame has unwound).
FRAME_OWNED_BY_FRAME_OBJECT = 0x04,
// The frame is a sentinel frame for entry to the interpreter loop
FRAME_OWNED_BY_INTERPRETER = 0x08,
};
struct _PyInterpreterFrameCore {
_PyStackRef f_executable; /* Deferred or strong reference (code object or None) */
struct _PyInterpreterFrameCore *previous;
char owner;
};
struct _PyInterpreterFrame {
_PyInterpreterFrameCore core;
....
};
typedef struct _PyInterpreterFrame * (*_PyFrame_Reifier)(struct _PyInterpreterFrameCore *, PyObject *reifier);
PyAPI_FUNC(PyObject *) PyUnstable_MakeExternalExecutable(_PyFrame_Reifier reifier, PyCodeObject *code, PyObject *state);
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
#105727
faster-cpython/ideas#575