context_switch2 : nr_switches is non monotonic across {(0,1), (0,2), (0, 8)} on Big-cores

[gautshen]

The thread-numbering on POWER9 big-core is an interleaving of the threads of the constituent small cores, i.e thread-ids {0, 2, 4, 6} belong to the same small core, while thread-ids {1, 3, 5, 7} belong to the other small core in the big-core.

The number of context-switches as observed with the context_switch2 benchmark (https://ozlabs.org/~anton/junkcode/context_switch2.c) for

• (0, 1) (two different small-cores of the same big-core)
• (0, 2) (same small core)
• (0, 8) (two different big-cores)
  are as follows:

# ./context_switch2 0 1 --timeout=5
354950
354112
353290
349704
350604

# ./context_switch2 0 2 --timeout=5
269772
269692
269420
269878
269702

# ./context_switch2 0 8 --timeout=5
289174
287990
288556
288812
288678

Thus the number of context-switches with (0,2) is lesser than those with (0,1) and those with (0,8). Understand the reason for this behaviour.

[gautshen]

With echo NO_TTWU_QUEUE > /sys/kernel/debug/sched_features we see an improvement for (0,1) and (0,2) while the nr-switches for (0,8) reduces. Thereby getting rid of the non-monotonicity that was observed earlier.

# ./context_switch2 0 1 --timeout=5
365862
366362
366156
365120
366626
# ./context_switch2 0 2 --timeout=5
278960
279166
279088
278988
279096
# ./context_switch2 0 8 --timeout=5
276906
277006
277272
277552
277332

[gautshen]

With TTWU_QUEUE enabled

cat /sys/kernel/debug/sched_features 
GENTLE_FAIR_SLEEPERS START_DEBIT NO_NEXT_BUDDY LAST_BUDDY CACHE_HOT_BUDDY WAKEUP_PREEMPTION NO_HRTICK NO_DOUBLE_TICK NONTASK_CAPACITY TTWU_QUEUE NO_SIS_AVG_CPU SIS_PROP NO_WARN_DOUBLE_CLOCK RT_PUSH_IPI RT_RUNTIME_SHARE NO_LB_MIN ATTACH_AGE_LOAD WA_IDLE WA_WEIGHT WA_BIAS UTIL_EST UTIL_EST_FASTUP 

# ./run_test.sh 8 9 5
312264
310790
311870
309510
310080
[ perf record: Woken up 357 times to write data ]
[ perf record: Captured and wrote 89.561 MB perf.data.8-9 (365910 samples) ]
 =========================== IPI details ================================
CPU 8
		  16 [XIVE-IPI 0 Edge IPI] = 146934 times
CPU 9
		  16 [XIVE-IPI 0 Edge IPI] = 218105 times

=================== Scheduler IPI events ===========================

Event smp_send_reschedule was hit 353467 times


Event arch_send_call_function_single_ipi was hit 12443 times

====================== snooze residency Details ============================

CPU 8:
State snooze : Disabled = 0: Usage = 431557 times(280593 us)
State CEDE : Disabled = 0: Usage = 11 times(39992 us)

CPU 9:
State snooze : Disabled = 0: Usage = 535213 times(280598 us)
State CEDE : Disabled = 0: Usage = 16 times(63111 us)

In this case we can notice that the number of IPIs sent across are pretty high (365039 in 5 seconds). Of these ~96% were in the form of smp_send_reschedule and remaining 4% were arch_send_call_function_single_ipi.

# ./run_test.sh 8 10 5
238044
238946
238976
246642
247182
[ perf record: Woken up 252 times to write data ]
[ perf record: Captured and wrote 63.290 MB perf.data.8-10 (258486 samples) ]
 =========================== IPI details ================================
CPU 8
		  16 [XIVE-IPI 0 Edge IPI] = 141336 times
CPU 10
		  16 [XIVE-IPI 0 Edge IPI] = 116230 times

=================== Scheduler IPI events ===========================

Event smp_send_reschedule was hit 171979 times


Event arch_send_call_function_single_ipi was hit 86507 times

====================== snooze residency Details ============================

CPU 8:
State snooze : Disabled = 0: Usage = 322842 times(385208 us)
State CEDE : Disabled = 0: Usage = 4 times(15968 us)

CPU 10:
State snooze : Disabled = 0: Usage = 359892 times(435873 us)
State CEDE : Disabled = 0: Usage = 106 times(852803 us)

In this case, the number of IPIs sent across were fewer (257566 in 5 seconds ) than the 8 9 case (365039 in 5 seconds). However the distribution is different. Only 67% of the IPIs were sent via smp_send_reschedule while the remaining 33% were sent via arch_send_call_function_single_ipi. The overall time spent in snooze is higher in the 8 10 case compared to 8 9 case. Thus it would perhaps have a better chance of being inside the snooze_loope withTIF_POLLING flag set. Which could perhaps explain the lower number of IPIs.

However the number of context switches in the 8 10 case drop by 20% compared to the 8 9 case. Wonder if it has anything to do with L1 flush, since in the 8 10 case, both the threads share the L1.

# ./run_test.sh 8 16 5
295738
295384
295616
295340
295504
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.248 MB perf.data.8-16 (423 samples) ]
 =========================== IPI details ================================
CPU 8
		  16 [XIVE-IPI 0 Edge IPI] = 233 times
CPU 16
		  16 [XIVE-IPI 0 Edge IPI] = 126 times

=================== Scheduler IPI events ===========================

Event smp_send_reschedule was hit 2 times


Event arch_send_call_function_single_ipi was hit 421 times

====================== snooze residency Details ============================

CPU 8:
State snooze : Disabled = 0: Usage = 738314 times(729878 us)
State CEDE : Disabled = 0: Usage = 11 times(43899 us)

CPU 16:
State snooze : Disabled = 0: Usage = 738417 times(732172 us)
State CEDE : Disabled = 0: Usage = 23 times(403752 us)

The number of IPIs are very minimal. This is due to the fact that the residency in snooze state is close to 2x compared to the 8 9 and 8 10 cases. Thus it is more likely that the waker CPU finds the wakee CPU in snooze_loop with TIF_POLLING flag set, and hence, desists from firing an IPI.