Introduce XPU scope profiler extending existing XPU profiler plugin #1174

moksiuc · 2025-11-13T09:04:28Z

Summary:

As XPU became a PyTorch built-in device, the profiler support is indispensable part of functionality completeness. In this PR, the XPU scope profiler is introduced by extending existing XPU profiler plugin. The XPU scope profiler is built on the foundation of intel PTI toolkit (https://github.com/intel/pti-gpu), and underlying SYCL runtime. It allows to gather XPU hardware metrics. The LIBKINETO_NOXPUPTI option is used to enable or disable the whole XPU profiler plugin during kineto build stage.

Changes:

Added new ActivityType : XPU_SCOPE_PROFILER, enabling the new scope profiler
Added new class XpuptiScopeProfilerConfig derived from AbstractConfig for configuration of the new scope profiler
Enhanced ChromeTraceLogger::handleActivity method so it outputs XPU hardware metrics from the new scope profiler in Perfetto counters display mode ("C")
Added gtest

moksiuc · 2025-11-13T09:28:50Z

@EikanWang, @gujinghui

libkineto/src/plugin/xpupti/FindSYCLToolkit.cmake

gujinghui · 2025-11-14T06:34:11Z

@moksiuc It's great that we are going to update our PTI integration code, and introduce new profiler path.
Could you help address below questions?

The alternative of ScopeProfiler is the RangeProfiler for CUDA? Looks like the RangeProfler is not enabled in PyTorch by default so far. Do you know why?
This PR is too huge to review. Can we split it to several PRs? For example, one PR for code refactor or cleanup per kineto or PTI changes, one or two PRs for ScopeProfiler, one PR for ChromeTraceLogger enhancement, and add test cases for each PRs.
BTW, CUDA provides CUDA_DRIVER activity to trace the driver actions. We should provide L0 actions as the counterpart, right? I remember, PTI should be able to do that. Do we have plan to cover it?

kineto/libkineto/src/ActivityType.cpp

Line 29 in 1e30d37

{"cuda_driver", ActivityType::CUDA_DRIVER},

moksiuc · 2025-11-14T13:53:03Z

The alternative of ScopeProfiler is the RangeProfiler for CUDA? Looks like the RangeProfler is not enabled in PyTorch by default so far. Do you know why?
It is enabled by providing experimental_config=_ExperimentalConfig(...). I don't know why it is this way but we are enabling our profiler the same way. One of the reasons may be that Range/Scope profiler requires parameters like HW metrics names that are passed through _ExperimentalConfig.

moksiuc · 2025-11-14T13:53:59Z

BTW, CUDA provides CUDA_DRIVER activity to trace the driver actions. We should provide L0 actions as the counterpart, right? I remember, PTI should be able to do that. Do we have plan to cover it?

kineto/libkineto/src/ActivityType.cpp

Line 29 in 1e30d37

{"cuda_driver", ActivityType::CUDA_DRIVER},

For sure not in this PR. I'll add this to our list of tasks.

moksiuc · 2025-11-17T10:35:03Z

This PR is too huge to review. Can we split it to several PRs? For example, one PR for code refactor or cleanup per kineto or PTI changes, one or two PRs for ScopeProfiler, one PR for ChromeTraceLogger enhancement, and add test cases for each PRs.

Extracted clean up and adding config for scope profiler to separate PR's.
This one should be much smaller afterwards.
Currently I don't see further areas of extracting separate PR's as what would remain is full scope profiler implementation with tests and we'd like not to introduce half of the implementation that is not working functionally.

- removed rangeEnabled - fix test to align to this removal - erase used kernelActivity from map

- place of config initialization - removal of passing unused C compiler flag into test cmake file

gujinghui

This PR is split to #1177, #1180, and more.

gujinghui · 2025-12-02T01:51:47Z

@moksiuc Let's close this PR.

moksiuc · 2025-12-03T07:48:58Z

@gujinghui this is the core of the scope profiler. When 2 smaller parts are merged this one would have only core profiler left.

Otherwise old profiler requires setting ZET_ENABLE_METRICS what is incorrect requirement.

moksiuc · 2025-12-19T11:27:38Z

@EikanWang, @gujinghui
Please review.
2 extracted parts are already merged.
Now this change contains only code directly related to scope profiler.

moksiuc · 2025-12-19T11:28:31Z

Tests are run on pytorch part:
pytorch/pytorch#165766

libkineto/src/plugin/xpupti/XpuptiActivityApi.cpp

libkineto/test/xpupti/compute/CMakeLists.txt

gujinghui

LGTM. I assume you already verified it on local real machine, right? @moksiuc

moksiuc · 2025-12-22T10:29:17Z

LGTM. I assume you already verified it on local real machine, right? @moksiuc

Yes, I did.

gujinghui · 2026-01-06T01:23:02Z

@malfet @sraikund16 could you help review this PR? Thanks.

moksiuc · 2026-01-12T08:44:21Z

@malfet @sraikund16 could you review, please ?

divyanshk

Thanks for the PR. I'm ramping up on this space. I'm curious to understand how much does it differ from the existing XPU profiler - maybe it should exist as a separate class in itself ? How can I learn more about the scope profiler ?

Also, thank you for adding tests.

libkineto/include/ITraceActivity.h

divyanshk · 2026-01-12T19:00:41Z

libkineto/src/plugin/xpupti/XpuptiActivityApi.cpp

 #endif

-void XpuptiActivityApi::enableXpuptiActivities(
+#if PTI_VERSION_AT_LEAST(0, 15)


Nit: XPUPTI_VERSION_AT_LEAST would be a better name for this macro. I can update that later (not relevant for this PR).

Ok, thanks for hint. I can update it if I have to make any other update in the code and push new commit.

After thinking about it XPUPTI_VERSION_AT_LEAST might be misleading as it is not xpupti version (that is part of kineto) but the PTI component version that xpupti uses underneath. Actually there is whole PTI api referenced starting with pti* prefix and PTI_VERSION_AT_LEAST refers to version of that api.

moksiuc · 2026-01-13T08:57:26Z

Thanks for the PR. I'm ramping up on this space. I'm curious to understand how much does it differ from the existing XPU profiler - maybe it should exist as a separate class in itself ? How can I learn more about the scope profiler ?

Also, thank you for adding tests.

This PR adds new activity ActivityType::XPU_SCOPE_PROFILER, accepted by XPU profiler. With this new mode enabled XPU profiler additionally gathers requested HW metrics.
I don't think the separate class is necessary as it just adds additional mode to a few existing so far.
The new mode is a XPU alternative for what is implemented in class CuptiRangeProfiler.

gujinghui · 2026-01-14T07:29:04Z

Thanks for the PR. I'm ramping up on this space. I'm curious to understand how much does it differ from the existing XPU profiler - maybe it should exist as a separate class in itself ? How can I learn more about the scope profiler ?
Also, thank you for adding tests.

This PR adds new activity ActivityType::XPU_SCOPE_PROFILER, accepted by XPU profiler. With this new mode enabled XPU profiler additionally gathers requested HW metrics. I don't think the separate class is necessary as it just adds additional mode to a few existing so far. The new mode is a XPU alternative for what is implemented in class CuptiRangeProfiler.

Hi @divyanshk ,

For more details of CuptiRangeProfiler, please see this page. https://deepwiki.com/pytorch/kineto/3.1-nvidia-gpu-support-(cupti)#4-cupti-range-profiler-integration

@moksiuc is going to provide similar functionalities on XPU in this PR.

scotts · 2026-01-15T15:20:28Z

Hi, @moksiuc, I'm also a new maintainer to this library. Thanks for all of the recent cleanup PRs!

On this PR, I have a similar concern that @divyanshk does on the code structuring. Specifically, this code uses #ifdefs to conditionally add member functions, member functions and member structs. This library has a history of doing that, but we want to start untangling that as it's difficult to maintain.

We should be able to achieve the same functionality by defining XpuptiActivityScopeApi (or a similar name) that derives from XpuptiActivityApi. Then we can use the #ifdefs to guard the entire files, which is much easier to reason about. You may need to still modify the XpuptiActivityApi interface itself a little so that it all works cleanly with XpuActivityProfiler, but that seems reasonable.

scope profiler squashed

52c714a

meta-cla bot added the cla signed label Nov 13, 2025

moksiuc changed the title ~~scope profiler squashed~~ Introduce XPU scope profiler extending existing XPU profiler plugin Nov 13, 2025

Type fix

0cbe479

gujinghui reviewed Nov 14, 2025

View reviewed changes

libkineto/src/plugin/xpupti/FindSYCLToolkit.cmake Outdated Show resolved Hide resolved

This was referenced Nov 14, 2025

Code cleanup of XPU profiler for incoming scope profiler #1177

Closed

Add config for incoming XPU scope profiler #1180

Closed

moksiuc added 2 commits November 17, 2025 13:20

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

7c98f84

Fixes

8438227

- removed rangeEnabled - fix test to align to this removal - erase used kernelActivity from map

gujinghui approved these changes Nov 18, 2025

View reviewed changes

moksiuc added 3 commits November 18, 2025 15:29

Code review fixes

fd2a6e1

- place of config initialization - removal of passing unused C compiler flag into test cmake file

Fix clang build on PT level

88082d5

Fix clang build on PT level 2nd approach

04f305c

gujinghui suggested changes Nov 19, 2025

View reviewed changes

moksiuc added 5 commits November 19, 2025 12:41

lintrunner

dbc709f

Improve json for presentation in perfetto

ad4b1dd

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

e050a30

Fix gtest after json update

37eafea

Put correct oneapi version in error msg

a266b04

moksiuc marked this pull request as ready for review November 24, 2025 10:01

Fix correct PTI and OneApi version

4a70122

sraikund16 and others added 2 commits December 17, 2025 13:19

lint

535b19e

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

b639a8c

moksiuc and others added 6 commits December 17, 2025 13:21

Fix merge error

e85fd26

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

115ae6c

Move device gathering to enableScopeProfiler

02b8684

Otherwise old profiler requires setting ZET_ENABLE_METRICS what is incorrect requirement.

lint

371c860

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

9e3f9fd

Resolve exception from desctuctor

0d46ae9

moksiuc requested a review from gujinghui December 19, 2025 11:28

gujinghui reviewed Dec 22, 2025

View reviewed changes

libkineto/src/plugin/xpupti/XpuptiActivityApi.cpp Outdated Show resolved Hide resolved

gujinghui reviewed Dec 22, 2025

View reviewed changes

libkineto/test/xpupti/compute/CMakeLists.txt Show resolved Hide resolved

gujinghui approved these changes Dec 22, 2025

View reviewed changes

moksiuc added 4 commits December 22, 2025 13:58

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

906085b

Fix Win build: error C2039: 'back_inserter': is not a member of 'std'

898ce26

Update min PTI version error msg

a014058

Remove unnecesary includes

3fe5339

gujinghui approved these changes Dec 24, 2025

View reviewed changes

divyanshk reviewed Jan 12, 2026

View reviewed changes

moksiuc requested a review from divyanshk January 13, 2026 09:04

moksiuc added 2 commits January 15, 2026 15:42

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

75cd8f4

Avoid adding getMetadata method to the interface

b2c9521

ZhaoqiongZ added this to PyTorch Intel Jan 16, 2026

ZhaoqiongZ moved this to Aged Pending Review in PyTorch Intel Jan 16, 2026

ZhaoqiongZ moved this from Aged Pending Review to In Progress in PyTorch Intel Jan 16, 2026

Introduce XPU scope profiler extending existing XPU profiler plugin #1174

Are you sure you want to change the base?

Introduce XPU scope profiler extending existing XPU profiler plugin #1174

Uh oh!

Conversation

moksiuc commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moksiuc commented Nov 13, 2025

Uh oh!

Uh oh!

gujinghui commented Nov 14, 2025

Uh oh!

moksiuc commented Nov 14, 2025

Uh oh!

moksiuc commented Nov 14, 2025

Uh oh!

moksiuc commented Nov 17, 2025

Uh oh!

gujinghui left a comment

Choose a reason for hiding this comment

Uh oh!

gujinghui commented Dec 2, 2025

Uh oh!

moksiuc commented Dec 3, 2025

Uh oh!

moksiuc commented Dec 19, 2025

Uh oh!

moksiuc commented Dec 19, 2025

Uh oh!

Uh oh!

Uh oh!

gujinghui left a comment

Choose a reason for hiding this comment

Uh oh!

moksiuc commented Dec 22, 2025

Uh oh!

gujinghui commented Jan 6, 2026

Uh oh!

moksiuc commented Jan 12, 2026

Uh oh!

divyanshk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

divyanshk Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

moksiuc Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

moksiuc Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

moksiuc commented Jan 13, 2026

Uh oh!

gujinghui commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scotts commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

moksiuc commented Nov 13, 2025 •

edited

Loading

divyanshk left a comment •

edited

Loading

gujinghui commented Jan 14, 2026 •

edited

Loading