CoreSight currently models exception entry by changing the preceding instruction range into a taken branch. This can give an IRQ the source PC of an instruction that already retired and overwrite a real branch immediately before the exception.
For an untaken B.LS followed by an IRQ, perf script currently reports:
hw int 4000f8 => ffff800080010c80 b.ls #0x400118 ... iret ffff800080012284 => 4000fc eret
The hardware trace supplies 0x4000fc as the preferred return address. For an IRQ, this is the architectural resume PC (Arm ARM, R_VBQMV). B.LS has retired, and the saved PC identifies the boundary before the following MOV. Using this PC as the IRQ source reflects the architectural state at exception entry:
hw int 4000fc => ffff800080010c80 movz x2, #0x1796 ... iret ffff800080012284 => 4000fc eret
The IRQ sample represents the transfer from this architectural execution position to the handler. The series synthesizes exception entries from exception packets, preserving the preceding branch and its outcome.
The supporting changes:
- Let decoders supply sample.ret_addr so later instruction fetching cannot change thread-stack return addresses. Apply this to Intel PT asynchronous samples as well. - Prepare packet ISA and instruction-size handling and share sample synthesis helpers. Mark untaken branches and break history when instruction memory is unavailable. - Add a thread-stack regression test and four AArch64 CoreSight tests. IRQs and page faults must resume at the entry PC; SVC and emulated MRS must resume four bytes later.
Based on the AI search and test on my x86 machine, this matches perf's Intel PT handling of IRQs. Intel PT records the next instruction's IP in the FUP packet, and Perf uses that IP as the interrupt sample's source.
This series is verified on Orion6 board with "perf test coresight".
Signed-off-by: Leo Yan leo.yan@arm.com --- Changes in v3: - Introduced the pref_ret_addr/end_addr union (James). - Clarified zero-size A32/A64 exception handling and why T32 retains per-address instruction decoding (James). - Retreed missing exception pairs for up to three recordings, then fail instead of skipping (James). - Shorten the IRQ workload (James). - Fix FIFO argument quoting and remove redundant ShellCheck source directives (James). - Checked instruction length before copying bytes into sample->insn (Sashiko). - Retried MRS checks when instruction page faults may match the opcode filter (Sashiko). - Rebased onto the updated perf tree and collect review tags. - Link to v2: https://lore.kernel.org/r/20260923-perf_cs_etm_fix_non_taken-v2-0-6ab8c07a54...
Changes in v2:
- Rework the fix around exception packets to preserve both exception entries and preceding branches. - Add explicit return addresses and the Intel PT asynchronous-branch fix. - Split out packet/synthesis preparation, record not-taken branches and handle unreadable instruction memory. - Add thread-stack regression coverage and four CoreSight tests. - Link to v1: https://lore.kernel.org/r/20260713-perf_cs_etm_fix_non_taken-v1-0-4561607fc6...
--- Leo Yan (14): perf sample: Allow decoders to supply branch return addresses perf intel-pt: Preserve return addresses for asynchronous branches perf cs-etm: Break branch history when instruction memory is unavailable perf cs-etm: Centralize packet ISA initialization perf cs-etm: Use the recorded instruction size for A32 and A64 perf cs-etm: Mark branches that were not taken perf cs-etm: Factor out final instruction sample synthesis perf cs-etm: Centralize branch sample synthesis checks perf cs-etm: Classify exception calls using the exception packet perf cs-etm: Synthesize exception entries separately from branches perf tests: Check CoreSight IRQ entry and exit perf tests: Check CoreSight syscall entry and exit perf tests: Check CoreSight abort entry and exit perf tests: Check CoreSight emulated instruction entry and exit
tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 3 + .../perf/tests/shell/coresight/abort_entry_exit.sh | 32 +++ tools/perf/tests/shell/coresight/irq_entry_exit.sh | 36 +++ .../tests/shell/coresight/syscall_entry_exit.sh | 32 +++ .../perf/tests/shell/coresight/trap_entry_exit.sh | 39 ++++ tools/perf/tests/shell/lib/coresight_exception.sh | 164 ++++++++++++++ tools/perf/tests/tests.h | 3 + tools/perf/tests/thread-stack.c | 106 +++++++++ tools/perf/tests/workloads/Build | 4 + tools/perf/tests/workloads/branch_not_taken_loop.c | 33 +++ tools/perf/tests/workloads/page_fault_loop.c | 37 ++++ tools/perf/util/cs-etm-decoder/cs-etm-decoder.c | 106 ++++++--- tools/perf/util/cs-etm.c | 246 +++++++++++---------- tools/perf/util/cs-etm.h | 6 +- tools/perf/util/intel-pt.c | 7 + tools/perf/util/sample.c | 1 + tools/perf/util/sample.h | 5 + tools/perf/util/thread-stack.c | 5 +- 19 files changed, 714 insertions(+), 152 deletions(-) --- base-commit: 4626d370ae89b8af4d46a45372b269f926ec662d change-id: 20260713-perf_cs_etm_fix_non_taken-5b4d7f73f41e
Best regards,
The thread stack derives return addresses from IP + insn_len. For an interrupt or fault, the return address can instead be the sample IP even when instruction bytes at that address are available.
Add ret_addr to perf_sample and prefer it in thread_stack__trace_end() and thread_stack__process() when supplied. Initialize it to zero so other samples retain the IP + insn_len calculation.
Add a regression test for explicit exception return addresses and the ordinary call fallback when the return address is zero.
Reviewed-by: Adrian Hunter adrian.hunter@intel.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/tests.h | 1 + tools/perf/tests/thread-stack.c | 106 ++++++++++++++++++++++++++++++++++++++++ tools/perf/util/sample.c | 1 + tools/perf/util/sample.h | 5 ++ tools/perf/util/thread-stack.c | 5 +- 7 files changed, 118 insertions(+), 2 deletions(-)
diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build index 58858f3d2bf954e125182d9f326817c69772f70a..5eebb2ec5ddefcfda0efe5c08e5c4e3ebf75e707 100644 --- a/tools/perf/tests/Build +++ b/tools/perf/tests/Build @@ -30,6 +30,7 @@ perf-test-y += task-exit.o perf-test-y += sw-clock.o perf-test-y += mmap-thread-lookup.o perf-test-y += thread-maps-share.o +perf-test-y += thread-stack.o perf-test-$(CONFIG_LIBTRACEEVENT) += switch-tracking.o perf-test-y += keep-tracking.o perf-test-y += code-reading.o diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c index 0b8897fd42cace9381981c1e2cf5402be0026c90..8dd72cd97cdb2de0f80f108cd72b9264c073e2a0 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -132,6 +132,7 @@ static struct test_suite *generic_tests[] = { &suite__hists_filter, &suite__mmap_thread_lookup, &suite__thread_maps_share, + &suite__thread_stack, &suite__hists_output, &suite__hists_cumulate, #ifdef HAVE_LIBTRACEEVENT diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h index 20ae3609892b612e92bc0881e301d4f295b764ec..59b66fe34a71420715ca8e04bf5f0f181b2c9cf9 100644 --- a/tools/perf/tests/tests.h +++ b/tools/perf/tests/tests.h @@ -156,6 +156,7 @@ DECLARE_SUITE(expr); DECLARE_SUITE(hists_filter); DECLARE_SUITE(mmap_thread_lookup); DECLARE_SUITE(thread_maps_share); +DECLARE_SUITE(thread_stack); DECLARE_SUITE(hists_output); DECLARE_SUITE(hists_cumulate); DECLARE_SUITE(switch_tracking); diff --git a/tools/perf/tests/thread-stack.c b/tools/perf/tests/thread-stack.c new file mode 100644 index 0000000000000000000000000000000000000000..0239e3bee34faf8fd4ddc7b723cf479609760064 --- /dev/null +++ b/tools/perf/tests/thread-stack.c @@ -0,0 +1,106 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/kernel.h> +#include <unistd.h> +#include "tests.h" +#include "util/addr_location.h" +#include "util/event.h" +#include "util/sample.h" +#include "util/thread.h" +#include "util/thread-stack.h" + +#define CALL_REF 1234UL +#define RET_REF 5678UL + +struct return_check { + unsigned int matched; + unsigned int unmatched; +}; + +static int check_call_return(struct call_return *cr, + u64 *parent_db_id __maybe_unused, void *data) +{ + struct return_check *check = data; + + if (cr->call_ref == CALL_REF && cr->return_ref == RET_REF && !cr->flags) + check->matched++; + else + check->unmatched++; + + return 0; +} + +/* A zero expected_ret_addr asks the stack to use ip + insn_len. */ +static int check_return_address(u64 expected_ret_addr, u64 actual_ret_addr, + u32 flags) +{ + struct call_return_processor *crp; + struct return_check check = { }; + struct thread *thread; + struct addr_location from = { }, to = { }; + struct perf_sample sample = { }; + int ret = TEST_FAIL; + + thread = thread__new(getpid(), getpid()); + if (!thread) + return TEST_FAIL; + + crp = call_return_processor__new(check_call_return, &check); + if (!crp) + goto out; + + sample.ip = 0x1000; /* Call or exception source addr */ + sample.addr = 0x2000; /* Callee or exception handler addr */ + sample.ret_addr = expected_ret_addr; + sample.flags = PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_CALL | flags; + sample.time = 1; + /* Model the opcode length after an instruction fetch. */ + sample.insn_len = 4; + if (thread_stack__process(thread, thread__comm(thread), &sample, + &from, &to, CALL_REF, crp)) + goto out; + + sample.ip = 0x2000; /* Return instruction addr */ + sample.addr = actual_ret_addr; /* Return branch target addr */ + sample.ret_addr = 0; + sample.flags = PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_RETURN | + (flags & (PERF_IP_FLAG_INTERRUPT | PERF_IP_FLAG_SYSCALLRET)); + sample.time = 2; + if (thread_stack__process(thread, thread__comm(thread), &sample, + &to, &from, RET_REF, crp)) + goto out; + + if (check.matched == 1 && !check.unmatched) + ret = TEST_OK; + +out: + thread__put(thread); + call_return_processor__free(crp); + return ret; +} + +static int test__thread_stack(struct test_suite *test __maybe_unused, + int subtest __maybe_unused) +{ + static const struct { + const char *name; + u64 expected_ret_addr; + u64 actual_ret_addr; + u32 flags; + } cases[] = { + { "ordinary call", 0, 0x1004, 0 }, + { "interrupt", 0x1000, 0x1000, PERF_IP_FLAG_ASYNC | PERF_IP_FLAG_INTERRUPT }, + { "fault or trap", 0x1000, 0x1000, PERF_IP_FLAG_INTERRUPT }, + { "SVC", 0x1004, 0x1004, PERF_IP_FLAG_SYSCALLRET }, + }; + + for (size_t i = 0; i < ARRAY_SIZE(cases); i++) { + if (check_return_address(cases[i].expected_ret_addr, + cases[i].actual_ret_addr, cases[i].flags)) { + pr_debug("Incorrect return address for %s\n", cases[i].name); + return TEST_FAIL; + } + } + return TEST_OK; +} + +DEFINE_SUITE("Thread stack return addresses after instruction fetching", thread_stack); diff --git a/tools/perf/util/sample.c b/tools/perf/util/sample.c index bccc19e2aaf25118a8ecde88473aa6cb16fa561a..4abb689132bc27d0f2318eeafca20f7e57621837 100644 --- a/tools/perf/util/sample.c +++ b/tools/perf/util/sample.c @@ -29,6 +29,7 @@ void perf_sample__init(struct perf_sample *sample, bool all) sample->intr_regs = NULL; sample->merged_callchain = false; sample->callchain = NULL; + sample->ret_addr = 0; } }
diff --git a/tools/perf/util/sample.h b/tools/perf/util/sample.h index cb4b16654876e9a5c3bfd66be3e0235ac15795e0..865ed18200f1339f5949ef3f185ed57a34aace72 100644 --- a/tools/perf/util/sample.h +++ b/tools/perf/util/sample.h @@ -131,6 +131,11 @@ struct perf_sample { u64 time; /** @addr: The sample event PERF_SAMPLE_ADDR value. */ u64 addr; + /** + * @ret_addr: Return address supplied by the decoder for a branch sample. + * Zero means use ip + insn_len. + */ + u64 ret_addr; /** @id: The sample event PERF_SAMPLE_ID or PERF_SAMPLE_IDENTIFIER value. */ u64 id; /** @stream_id: The sample event PERF_SAMPLE_STREAM_ID value. */ diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c index 1360f44421ef8bb80fe5cfdba59be7e6029b0240..d452d1a7eabb16f040ca3dada23e6fdec8fb1088 100644 --- a/tools/perf/util/thread-stack.c +++ b/tools/perf/util/thread-stack.c @@ -1030,7 +1030,7 @@ static int thread_stack__trace_end(struct thread_stack *ts, cp = call_path__findnew(cpr, ts->stack[ts->cnt - 1].cp, NULL, 0, ts->kernel_start);
- ret_addr = sample->ip + sample->insn_len; + ret_addr = sample->ret_addr ? sample->ret_addr : sample->ip + sample->insn_len;
return thread_stack__push_cp(ts, ret_addr, sample->time, ref, cp, false, true); @@ -1154,7 +1154,8 @@ int thread_stack__process(struct thread *thread, struct comm *comm, if (!sample->ip || !sample->addr) return 0;
- ret_addr = sample->ip + sample->insn_len; + /* Opcode fetching must not change a decoder-supplied return address. */ + ret_addr = sample->ret_addr ? sample->ret_addr : sample->ip + sample->insn_len; if (ret_addr == sample->addr) return 0; /* Zero-length calls are excluded */
Intel PT uses a zero instruction length for asynchronous branches so the thread stack resumes at the source IP. A dlfilter can fetch the instruction before call/return processing and replace that zero with its opcode length, advancing the saved return address.
Set sample.ret_addr to sample.ip for asynchronous branch samples so instruction fetching cannot change the return address. This also covers asynchronous trace-end samples and VM exits.
Reviewed-by: Adrian Hunter adrian.hunter@intel.com Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/intel-pt.c | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/tools/perf/util/intel-pt.c b/tools/perf/util/intel-pt.c index 8c21c9f52d572d2dd4e15b3b5c341b253ccea1b5..e8b9c8a67b0a0635e0110eda4b6ccb2511132fc7 100644 --- a/tools/perf/util/intel-pt.c +++ b/tools/perf/util/intel-pt.c @@ -1803,6 +1803,13 @@ static int intel_pt_synth_branch_sample(struct intel_pt_queue *ptq) perf_sample__init(&sample, /*all=*/true); intel_pt_prep_b_sample(pt, ptq, event, &sample);
+ /* + * For asynchronous branches, use the sample IP as the return address + * instead of advancing it during instruction decoding. + */ + if (sample.flags & PERF_IP_FLAG_ASYNC) + sample.ret_addr = sample.ip; + sample.id = ptq->pt->branches_id; sample.stream_id = ptq->pt->branches_id;
The decoder currently ignores OCSD_GEN_TRC_ELEM_ADDR_NACC, allowing a subsequent range to be paired with stale branch history from before the unreadable code.
Treat an inaccessible instruction address as a discontinuity and flush the pending range and branch history before processing subsequent elements.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm-decoder/cs-etm-decoder.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c index b56cf57a68c88de4686ecc8b0c7e71ac0f1d2f09..3fbb3fdf0382705bd4efa732c54ebb816144b2a8 100644 --- a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c +++ b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c @@ -594,7 +594,8 @@ static ocsd_datapath_resp_t cs_etm_decoder__gen_trace_elem_printer(
if (type == OCSD_GEN_TRC_ELEM_EO_TRACE || type == OCSD_GEN_TRC_ELEM_NO_SYNC || - type == OCSD_GEN_TRC_ELEM_TRACE_ON) + type == OCSD_GEN_TRC_ELEM_TRACE_ON || + type == OCSD_GEN_TRC_ELEM_ADDR_NACC) resp = cs_etm_decoder__buffer_discontinuity(etmq, packet_queue, trace_chan_id); else if (type == OCSD_GEN_TRC_ELEM_INSTR_RANGE)
Exception packets need their own ISA so the frontend can generate samples directly from them. Currently, ISA initialization is limited to instruction range packets.
Move ISA initialization into cs_etm_decoder__buffer_packet() and use a common helper to obtain it from the OpenCSD element. Populate the ISA for instruction ranges, context updates and exceptions, leaving it unknown for other element types.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm-decoder/cs-etm-decoder.c | 69 +++++++++++++++---------- 1 file changed, 41 insertions(+), 28 deletions(-)
diff --git a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c index 3fbb3fdf0382705bd4efa732c54ebb816144b2a8..911bb28db2985bc65828ad342ca5b0a8bb903e47 100644 --- a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c +++ b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c @@ -368,9 +368,38 @@ cs_etm_decoder__reset_timestamp(struct cs_etm_packet_queue *packet_queue) packet_queue->instr_count = 0; }
+static enum cs_etm_isa +cs_etm_decoder__get_isa(const ocsd_generic_trace_elem *elem) +{ + /* + * OpenCSD preserves ISA and context between output elements, including + * exceptions after a context change with no intervening range. + */ + if (elem->elem_type != OCSD_GEN_TRC_ELEM_PE_CONTEXT && + elem->elem_type != OCSD_GEN_TRC_ELEM_INSTR_RANGE && + elem->elem_type != OCSD_GEN_TRC_ELEM_EXCEPTION) + return CS_ETM_ISA_UNKNOWN; + + switch (elem->isa) { + case ocsd_isa_aarch64: + return CS_ETM_ISA_A64; + case ocsd_isa_arm: + return CS_ETM_ISA_A32; + case ocsd_isa_thumb2: + return CS_ETM_ISA_T32; + case ocsd_isa_tee: + case ocsd_isa_jazelle: + case ocsd_isa_custom: + case ocsd_isa_unknown: + default: + return CS_ETM_ISA_UNKNOWN; + } +} + static ocsd_datapath_resp_t cs_etm_decoder__buffer_packet(struct cs_etm_queue *etmq, struct cs_etm_packet_queue *packet_queue, + const ocsd_generic_trace_elem *elem, const u8 trace_chan_id, enum cs_etm_sample_type sample_type) { @@ -389,7 +418,7 @@ cs_etm_decoder__buffer_packet(struct cs_etm_queue *etmq, packet_queue->packet_count++;
packet_queue->packet_buffer[et].sample_type = sample_type; - packet_queue->packet_buffer[et].isa = CS_ETM_ISA_UNKNOWN; + packet_queue->packet_buffer[et].isa = cs_etm_decoder__get_isa(elem); packet_queue->packet_buffer[et].cpu = cpu; packet_queue->packet_buffer[et].start_addr = CS_ETM_INVAL_ADDR; packet_queue->packet_buffer[et].end_addr = CS_ETM_INVAL_ADDR; @@ -420,31 +449,13 @@ cs_etm_decoder__buffer_range(struct cs_etm_queue *etmq, int ret = 0; struct cs_etm_packet *packet;
- ret = cs_etm_decoder__buffer_packet(etmq, packet_queue, trace_chan_id, - CS_ETM_RANGE); + ret = cs_etm_decoder__buffer_packet(etmq, packet_queue, elem, + trace_chan_id, CS_ETM_RANGE); if (ret != OCSD_RESP_CONT && ret != OCSD_RESP_WAIT) return ret;
packet = &packet_queue->packet_buffer[packet_queue->tail];
- switch (elem->isa) { - case ocsd_isa_aarch64: - packet->isa = CS_ETM_ISA_A64; - break; - case ocsd_isa_arm: - packet->isa = CS_ETM_ISA_A32; - break; - case ocsd_isa_thumb2: - packet->isa = CS_ETM_ISA_T32; - break; - case ocsd_isa_tee: - case ocsd_isa_jazelle: - case ocsd_isa_custom: - case ocsd_isa_unknown: - default: - packet->isa = CS_ETM_ISA_UNKNOWN; - } - packet->start_addr = elem->st_addr; packet->end_addr = elem->en_addr; packet->instr_count = elem->num_instr_range; @@ -483,6 +494,7 @@ cs_etm_decoder__buffer_range(struct cs_etm_queue *etmq, static ocsd_datapath_resp_t cs_etm_decoder__buffer_discontinuity(struct cs_etm_queue *etmq, struct cs_etm_packet_queue *queue, + const ocsd_generic_trace_elem *elem, const uint8_t trace_chan_id) { /* @@ -490,7 +502,7 @@ cs_etm_decoder__buffer_discontinuity(struct cs_etm_queue *etmq, * reset time statistics. */ cs_etm_decoder__reset_timestamp(queue); - return cs_etm_decoder__buffer_packet(etmq, queue, trace_chan_id, + return cs_etm_decoder__buffer_packet(etmq, queue, elem, trace_chan_id, CS_ETM_DISCONTINUITY); }
@@ -502,7 +514,7 @@ cs_etm_decoder__buffer_exception(struct cs_etm_queue *etmq, { int ret = 0; struct cs_etm_packet *packet;
- ret = cs_etm_decoder__buffer_packet(etmq, queue, trace_chan_id, + ret = cs_etm_decoder__buffer_packet(etmq, queue, elem, trace_chan_id, CS_ETM_EXCEPTION); if (ret != OCSD_RESP_CONT && ret != OCSD_RESP_WAIT) return ret; @@ -516,9 +528,10 @@ cs_etm_decoder__buffer_exception(struct cs_etm_queue *etmq, static ocsd_datapath_resp_t cs_etm_decoder__buffer_exception_ret(struct cs_etm_queue *etmq, struct cs_etm_packet_queue *queue, + const ocsd_generic_trace_elem *elem, const uint8_t trace_chan_id) { - return cs_etm_decoder__buffer_packet(etmq, queue, trace_chan_id, + return cs_etm_decoder__buffer_packet(etmq, queue, elem, trace_chan_id, CS_ETM_EXCEPTION_RET); }
@@ -555,8 +568,8 @@ cs_etm_decoder__set_tid(struct cs_etm_queue *etmq, elem->context.exception_level, tid)) return OCSD_RESP_FATAL_SYS_ERR;
- ret = cs_etm_decoder__buffer_packet(etmq, packet_queue, trace_chan_id, - CS_ETM_CONTEXT); + ret = cs_etm_decoder__buffer_packet(etmq, packet_queue, elem, + trace_chan_id, CS_ETM_CONTEXT); if (ret != OCSD_RESP_CONT && ret != OCSD_RESP_WAIT) return ret;
@@ -597,7 +610,7 @@ static ocsd_datapath_resp_t cs_etm_decoder__gen_trace_elem_printer( type == OCSD_GEN_TRC_ELEM_TRACE_ON || type == OCSD_GEN_TRC_ELEM_ADDR_NACC) resp = cs_etm_decoder__buffer_discontinuity(etmq, packet_queue, - trace_chan_id); + elem, trace_chan_id); else if (type == OCSD_GEN_TRC_ELEM_INSTR_RANGE) resp = cs_etm_decoder__buffer_range(etmq, packet_queue, elem, trace_chan_id); @@ -606,7 +619,7 @@ static ocsd_datapath_resp_t cs_etm_decoder__gen_trace_elem_printer( trace_chan_id); else if (type == OCSD_GEN_TRC_ELEM_EXCEPTION_RET) resp = cs_etm_decoder__buffer_exception_ret(etmq, packet_queue, - trace_chan_id); + elem, trace_chan_id); else if (type == OCSD_GEN_TRC_ELEM_TIMESTAMP) resp = cs_etm_decoder__do_hard_timestamp(etmq, elem, trace_chan_id,
Use packet->last_instr_size in cs_etm__instr_size() instead of a hard-coded value of four. Ordinary A32/A64 instruction ranges are unchanged, since their instructions are always four bytes.
Prepare for synthesizing exception entries separately from the preceding instruction range. An exception packet may have last_instr_size = 0, meaning that no instruction completed at the exception source PC and the exception return address must not advance past that PC. The thread stack computes the return address as the source PC plus the instruction size, so returning four would incorrectly advance the return address by four bytes.
Keep per-address decoding for T32. A range can mix 16-bit and 32-bit instructions, so last_instr_size describes only the final instruction and cannot determine the size at an arbitrary address in the range.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index bdcf9481312943b2bbcade8f9688311853c138f2..9126f6327e98b67852b6a55980d67abb24de17fe 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -1391,8 +1391,7 @@ static inline int cs_etm__instr_size(struct cs_etm_queue *etmq, if (packet->isa == CS_ETM_ISA_T32) return cs_etm__t32_instr_size(etmq, tidq, packet, addr);
- /* Otherwise, 4-byte instruction size for A32/A64 */ - return 4; + return packet->last_instr_size; }
static inline u64 cs_etm__first_executed_instr(struct cs_etm_packet *packet) @@ -1572,6 +1571,15 @@ static void cs_etm__copy_insn(struct cs_etm_queue *etmq,
sample->insn_len = cs_etm__instr_size(etmq, tidq, packet, sample->ip);
+ /* + * The instruction length should always fit within sample->insn, + * but check it to prevent an out-of-bounds write. + */ + if (sample->insn_len > sizeof(sample->insn)) { + sample->insn_len = 0; + return; + } + cs_etm__frontend_mem_access(etmq, tidq, packet, sample->ip, sample->insn_len, (void *)sample->insn); }
An exception can transfer execution to the kernel after any instruction. When this creates a trace boundary, perf may attribute the trace-end sample to the last instruction in the range. If that instruction is an untaken branch, reporting only its branch type can make the transfer look like a taken branch.
Set PERF_IP_FLAG_NOT_TAKEN for branch ranges when last_instr_taken_branch is false. This records the decoded branch outcome independently of exception entry.
For a discontinuity without an exception element, perf script previously reported an untaken B.LS as:
tr end jcc 4000f8 => 0 b.ls #0x400118 tr strt jmp 0 => 4000fc
With the not-taken flag set:
tr end jcc/not_taken/ 4000f8 => 0 b.ls #0x400118 tr strt jmp 0 => 4000fc
Here, 4000f8 remains the last traced instruction and 4000fc is the trace restart address. These boundary samples do not establish an IRQ entry or an exception return; the not_taken flag records only the branch outcome.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm.c | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 9126f6327e98b67852b6a55980d67abb24de17fe..7cb9f69ead2b134b62653f133da7bdec23aaf280 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -2457,6 +2457,11 @@ static int cs_etm__set_sample_flags(struct cs_etm_queue *etmq, packet->flags = PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_RETURN;
+ /* Report branch that is not taken */ + if ((packet->flags & PERF_IP_FLAG_BRANCH) && + !packet->last_instr_taken_branch) + packet->flags |= PERF_IP_FLAG_NOT_TAKEN; + /* * Decoder might insert a discontinuity in the middle of * instruction packets, fixup prev_packet with flag
cs_etm__flush() and cs_etm__end_block() duplicate the instruction sample used to report the remaining branch history at the end of a trace.
Move this code into cs_etm__synth_last_instruction_sample(), preserving the sample generation and instruction period reset.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm.c | 71 ++++++++++++++++++------------------------------ 1 file changed, 26 insertions(+), 45 deletions(-)
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 7cb9f69ead2b134b62653f133da7bdec23aaf280..9abbbc28dd1e2a66eadd663ce18217cd62deb390 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -1727,6 +1727,28 @@ static int cs_etm__synth_instruction_sample(struct cs_etm_queue *etmq, return ret; }
+static int cs_etm__synth_last_instruction_sample(struct cs_etm_queue *etmq, + struct cs_etm_traceid_queue *tidq) +{ + struct cs_etm_packet *packet = tidq->prev_packet; + int ret; + + if (!etmq->etm->synth_opts.last_branch || + !etmq->etm->synth_opts.instructions) + return 0; + + if (packet->sample_type != CS_ETM_RANGE) + return 0; + + ret = cs_etm__synth_instruction_sample(etmq, tidq, packet, + cs_etm__last_executed_instr(packet), + tidq->period_instructions); + if (!ret) + tidq->period_instructions = 0; + + return ret; +} + /* * The cs etm packet encodes an instruction range between a branch target * and the next taken branch. Generate sample accordingly. @@ -2065,28 +2087,9 @@ static int cs_etm__flush(struct cs_etm_queue *etmq, if (tidq->prev_packet->sample_type == CS_ETM_EMPTY) goto swap_packet;
- if (etmq->etm->synth_opts.last_branch && - etmq->etm->synth_opts.instructions && - tidq->prev_packet->sample_type == CS_ETM_RANGE) { - u64 addr; - /* - * Generate a last branch event for the branches left in the - * circular buffer at the end of the trace. - * - * Use the address of the end of the last reported execution - * range - */ - addr = cs_etm__last_executed_instr(tidq->prev_packet); - - err = cs_etm__synth_instruction_sample( - etmq, tidq, tidq->prev_packet, addr, - tidq->period_instructions); - if (err) - return err; - - tidq->period_instructions = 0; - - } + err = cs_etm__synth_last_instruction_sample(etmq, tidq); + if (err) + return err;
if (etm->synth_opts.branches && tidq->prev_packet->sample_type == CS_ETM_RANGE) { @@ -2108,8 +2111,6 @@ static int cs_etm__flush(struct cs_etm_queue *etmq, static int cs_etm__end_block(struct cs_etm_queue *etmq, struct cs_etm_traceid_queue *tidq) { - int err; - /* * It has no new packet coming and 'etmq->packet' contains the stale * packet which was set at the previous time with packets swapping; @@ -2119,27 +2120,7 @@ static int cs_etm__end_block(struct cs_etm_queue *etmq, * event for the branches left in the circular buffer at the end of * the trace. */ - if (etmq->etm->synth_opts.last_branch && - etmq->etm->synth_opts.instructions && - tidq->prev_packet->sample_type == CS_ETM_RANGE) { - u64 addr; - - /* - * Use the address of the end of the last reported execution - * range. - */ - addr = cs_etm__last_executed_instr(tidq->prev_packet); - - err = cs_etm__synth_instruction_sample( - etmq, tidq, tidq->prev_packet, addr, - tidq->period_instructions); - if (err) - return err; - - tidq->period_instructions = 0; - } - - return 0; + return cs_etm__synth_last_instruction_sample(etmq, tidq); }
static int cs_etm__flush_stack_cb(struct thread *thread,
Move the branch-synthesis option and packet checks from cs_etm__sample() into cs_etm__synth_branch_sample(), alongside the existing branch filter. Call the helper directly from cs_etm__sample() and cs_etm__flush().
Accept trace-start and trace-end flags, this preserves boundary samples without a taken branch.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm.c | 37 ++++++++++++++----------------------- 1 file changed, 14 insertions(+), 23 deletions(-)
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 9abbbc28dd1e2a66eadd663ce18217cd62deb390..a9191c06060a2455e8bf1cd5f00fb9e92904eefa 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -1768,6 +1768,14 @@ static int cs_etm__synth_branch_sample(struct cs_etm_queue *etmq, } dummy_bs; u64 ip;
+ if (!etm->synth_opts.branches) + return 0; + + if (!cs_etm__packet_has_taken_branch(tidq->prev_packet) && + !(tidq->prev_packet->flags & (PERF_IP_FLAG_TRACE_BEGIN | + PERF_IP_FLAG_TRACE_END))) + return 0; + if (etm->branches_filter && !(etm->branches_filter & tidq->prev_packet->flags)) return 0; @@ -2007,23 +2015,9 @@ static int cs_etm__sample(struct cs_etm_queue *etmq, } }
- if (etm->synth_opts.branches) { - bool generate_sample = false; - - /* Generate sample for tracing on packet */ - if (tidq->prev_packet->sample_type == CS_ETM_DISCONTINUITY) - generate_sample = true; - - /* Generate sample for branch taken packet */ - if (cs_etm__packet_has_taken_branch(tidq->prev_packet)) - generate_sample = true; - - if (generate_sample) { - ret = cs_etm__synth_branch_sample(etmq, tidq); - if (ret) - return ret; - } - } + ret = cs_etm__synth_branch_sample(etmq, tidq); + if (ret) + return ret;
cs_etm__packet_swap(etm, tidq);
@@ -2091,12 +2085,9 @@ static int cs_etm__flush(struct cs_etm_queue *etmq, if (err) return err;
- if (etm->synth_opts.branches && - tidq->prev_packet->sample_type == CS_ETM_RANGE) { - err = cs_etm__synth_branch_sample(etmq, tidq); - if (err) - return err; - } + err = cs_etm__synth_branch_sample(etmq, tidq); + if (err) + return err;
swap_packet: cs_etm__packet_swap(etm, tidq);
ETMv4 and ETE use the same exception number for SVC, HVC and SMC. Identify SVC by inspecting the instruction before the preferred return address in the exception packet.
Preserve the exception level and preferred return address from OpenCSD, and use them with the packet's ISA in cs_etm__is_syscall() and cs_etm__is_sync_exception(). This makes exception call classification independent of the preceding instruction range.
Name the exception address pref_ret_addr in an anonymous union shared with the range end_addr, keeping the packet layout unchanged. Name the cs_etm__is_svc_instr() address argument ret_addr and document that it checks the instruction immediately before the return address.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm-decoder/cs-etm-decoder.c | 7 ++++++- tools/perf/util/cs-etm.c | 21 ++++++++++----------- tools/perf/util/cs-etm.h | 5 ++++- 3 files changed, 20 insertions(+), 13 deletions(-)
diff --git a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c index 911bb28db2985bc65828ad342ca5b0a8bb903e47..ccae10e6cdad55607e092b90a1a940103fa050d7 100644 --- a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c +++ b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c @@ -511,7 +511,8 @@ cs_etm_decoder__buffer_exception(struct cs_etm_queue *etmq, struct cs_etm_packet_queue *queue, const ocsd_generic_trace_elem *elem, const uint8_t trace_chan_id) -{ int ret = 0; +{ + int ret = 0; struct cs_etm_packet *packet;
ret = cs_etm_decoder__buffer_packet(etmq, queue, elem, trace_chan_id, @@ -521,6 +522,10 @@ cs_etm_decoder__buffer_exception(struct cs_etm_queue *etmq,
packet = &queue->packet_buffer[queue->tail]; packet->exception_number = elem->exception_number; + if (elem->context.el_valid) + packet->el = elem->context.exception_level; + if (elem->excep_ret_addr) + packet->pref_ret_addr = elem->en_addr;
return ret; } diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index a9191c06060a2455e8bf1cd5f00fb9e92904eefa..14a6466255017386addaee539141bf89b6dcf76f 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -2061,7 +2061,7 @@ static int cs_etm__exception(struct cs_etm_traceid_queue *tidq) * to generate branch sample for the instruction range before the * exception is trapped to kernel or before the exception returning. * - * The exception packet includes the dummy address values, so don't + * The exception packet does not describe an instruction range, so don't * swap PACKET with PREV_PACKET. This keeps PREV_PACKET to be useful * for generating instruction and branch samples. */ @@ -2192,9 +2192,10 @@ static int cs_etm__get_data_block(struct cs_etm_queue *etmq) return 1; }
+/* Check whether the instruction before ret_addr is an SVC. */ static bool cs_etm__is_svc_instr(struct cs_etm_queue *etmq, struct cs_etm_traceid_queue *tidq, - struct cs_etm_packet *packet, u64 end_addr) + struct cs_etm_packet *packet, u64 ret_addr) { /* Initialise to keep compiler happy */ u16 instr16 = 0; @@ -2215,7 +2216,7 @@ static bool cs_etm__is_svc_instr(struct cs_etm_queue *etmq, * with 16 bits instruction and has no definition for 32bits; * so below only read 2 bytes as instruction size for T32. */ - addr = end_addr - 2; + addr = ret_addr - 2; cs_etm__frontend_mem_access(etmq, tidq, packet, addr, sizeof(instr16), (u8 *)&instr16); if ((instr16 & 0xFF00) == 0xDF00) @@ -2231,7 +2232,7 @@ static bool cs_etm__is_svc_instr(struct cs_etm_queue *etmq, * | !1111 | 1 1 1 1 | imm24 | * +---------+---------+-------------------------+ */ - addr = end_addr - 4; + addr = ret_addr - 4; cs_etm__frontend_mem_access(etmq, tidq, packet, addr, sizeof(instr32), (u8 *)&instr32); if ((instr32 & 0x0F000000) == 0x0F000000 && @@ -2248,7 +2249,7 @@ static bool cs_etm__is_svc_instr(struct cs_etm_queue *etmq, * | 1 1 0 1 0 1 0 0 0 0 0 | imm16 | 0 0 0 0 1 | * +-----------------------+---------+-----------+ */ - addr = end_addr - 4; + addr = ret_addr - 4; cs_etm__frontend_mem_access(etmq, tidq, packet, addr, sizeof(instr32), (u8 *)&instr32); if ((instr32 & 0xFFE0001F) == 0xd4000001) @@ -2267,7 +2268,6 @@ static bool cs_etm__is_syscall(struct cs_etm_queue *etmq, struct cs_etm_traceid_queue *tidq, u64 magic) { struct cs_etm_packet *packet = tidq->packet; - struct cs_etm_packet *prev_packet = tidq->prev_packet;
if (magic == __perf_cs_etmv3_magic) if (packet->exception_number == CS_ETMV3_EXC_SVC) @@ -2280,8 +2280,8 @@ static bool cs_etm__is_syscall(struct cs_etm_queue *etmq, */ if (magic == __perf_cs_etmv4_magic || magic == __perf_cs_ete_magic) { if (packet->exception_number == CS_ETMV4_EXC_CALL && - cs_etm__is_svc_instr(etmq, tidq, prev_packet, - prev_packet->end_addr)) + cs_etm__is_svc_instr(etmq, tidq, packet, + packet->pref_ret_addr)) return true; }
@@ -2319,7 +2319,6 @@ static bool cs_etm__is_sync_exception(struct cs_etm_queue *etmq, u64 magic) { struct cs_etm_packet *packet = tidq->packet; - struct cs_etm_packet *prev_packet = tidq->prev_packet;
if (magic == __perf_cs_etmv3_magic) if (packet->exception_number == CS_ETMV3_EXC_SMC || @@ -2343,8 +2342,8 @@ static bool cs_etm__is_sync_exception(struct cs_etm_queue *etmq, * (SMC, HVC) are taken as sync exceptions. */ if (packet->exception_number == CS_ETMV4_EXC_CALL && - !cs_etm__is_svc_instr(etmq, tidq, prev_packet, - prev_packet->end_addr)) + !cs_etm__is_svc_instr(etmq, tidq, packet, + packet->pref_ret_addr)) return true;
/* diff --git a/tools/perf/util/cs-etm.h b/tools/perf/util/cs-etm.h index b81099c2b301c25f894cc9a50edfde87aceff49d..e40b9b7d0a9bd90f089c2f20527332264ead4249 100644 --- a/tools/perf/util/cs-etm.h +++ b/tools/perf/util/cs-etm.h @@ -174,7 +174,10 @@ struct cs_etm_packet { enum cs_etm_sample_type sample_type; enum cs_etm_isa isa; u64 start_addr; - u64 end_addr; + union { + u64 end_addr; /* Range packets */ + u64 pref_ret_addr; /* Exception packets */ + }; u32 instr_count; u32 last_instr_type; u32 last_instr_subtype;
Exception entry is currently represented by replacing the preceding range's flags and forcing its last_instr_taken_branch bit. This gives an interrupt the source address of the last executed instruction. When that instruction is a taken branch, its edge and original flags are lost.
For example, consider an untaken B.LS followed by an IRQ:
4000f4: eb02003f cmp x1, x2 4000f8: 54000109 b.ls 400118 4000fc: d282f2c2 mov x2, #0x1796
If B.LS completes untaken and the IRQ is taken before MOV completes, the preferred exception return address is 4000fc. With kernel tracing disabled, perf script previously attributed the IRQ to the preceding range:
tr end hw int 4000f8 => 0 b.ls #0x400118 tr strt jmp 0 => 4000fc
When an IRQ element provides that return address, use it as the source of the synthesized exception sample:
tr end hw int 4000fc => 0 movz x2, #0x1796 tr strt jmp 0 => 4000fc
The source now identifies the architectural resume PC, rather than the last completed instruction. The IRQ signal could have arrived while B.LS was executing. The traced preferred exception return address of 4000fc confirms that B.LS had retired architecturally before the IRQ was taken.
OpenCSD provides an exception's preferred return address in en_addr when excep_ret_addr is set. It does not define or initialize st_addr for an exception element. Derive A32 and A64 exception source PCs from the preferred return address. Exception calls (SVC/SMC/HVC) use the preceding four-byte instruction; interrupts, faults and traps use the return address with last_instr_size set to zero.
Use pref_ret_addr when deriving exception source PCs in the decoder and cs_etm__last_executed_instr(), and when setting sample.ret_addr.
Keep T32 exception samples on the existing fallback path because determining the call size requires an instruction read. Retain pref_ret_addr for the frontend's SVC check. RESET and missing return addresses also use the fallback because their source PC is unknown.
Resolve the preceding branch with its original flags without adding instructions. Then retain the exception in prev_packet until the next range, exception or discontinuity resolves its destination. This preserves both edges when an exception follows a taken branch and also handles consecutive exceptions.
Supply the preferred return address in sample.ret_addr so later instruction fetching cannot change the return PC used by call/return export. Generate final instruction samples only from nonempty instruction ranges.
CS_ETM_EXCEPTION_RET needs no further handling in the packet processing loop because ERET is already part of the preceding instruction range. cs_etm__set_sample_flags() now sets both that range's return flags and last_instr_taken_branch.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/util/cs-etm-decoder/cs-etm-decoder.c | 31 ++++++- tools/perf/util/cs-etm.c | 106 +++++++++++++++--------- tools/perf/util/cs-etm.h | 1 + 3 files changed, 95 insertions(+), 43 deletions(-)
diff --git a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c index ccae10e6cdad55607e092b90a1a940103fa050d7..b9397151748b222ae4542394c7b974b0fdb69bba 100644 --- a/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c +++ b/tools/perf/util/cs-etm-decoder/cs-etm-decoder.c @@ -514,6 +514,7 @@ cs_etm_decoder__buffer_exception(struct cs_etm_queue *etmq, { int ret = 0; struct cs_etm_packet *packet; + u8 last_instr_size = 0;
ret = cs_etm_decoder__buffer_packet(etmq, queue, elem, trace_chan_id, CS_ETM_EXCEPTION); @@ -524,8 +525,34 @@ cs_etm_decoder__buffer_exception(struct cs_etm_queue *etmq, packet->exception_number = elem->exception_number; if (elem->context.el_valid) packet->el = elem->context.exception_level; - if (elem->excep_ret_addr) - packet->pref_ret_addr = elem->en_addr; + + /* RESET has no defined preferred return address. */ + if (!elem->excep_ret_addr || + packet->exception_number == CS_ETMV4_EXC_RESET) + return ret; + + packet->pref_ret_addr = elem->en_addr; + + /* + * T32 calls can be two or four bytes, requiring an instruction read + * before pref_ret_addr to determine their size. Keep it for the + * frontend's SVC check and leave start_addr unknown for its fallback. + */ + if (packet->isa != CS_ETM_ISA_A32 && packet->isa != CS_ETM_ISA_A64) + return ret; + + /* + * For A32 and A64, exception calls execute a four-byte SVC, HVC or + * SMC and save the following PC. Other exceptions use the interrupted + * or faulting PC, including traps on those instructions. + * + * With a shared branch target, no instruction has completed + * at the target, so last_instr_size remains zero. + */ + if (elem->exception_number == CS_ETMV4_EXC_CALL) + last_instr_size = 4; + packet->start_addr = packet->pref_ret_addr - last_instr_size; + packet->last_instr_size = last_instr_size;
return ret; } diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 14a6466255017386addaee539141bf89b6dcf76f..c8a6a6e4c7d3b4c9d58f3a5b46587ee53c864b2e 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -101,10 +101,8 @@ struct cs_etm_traceid_queue { ocsd_ex_level decode_el;
/* - * The frontend accesses the EL from '[prev_]packet' because it needs - * previous EL for branch and current EL for instruction samples. It's - * not possible to change thread in a single branch sample so no need to - * store or access the thread through the packet. + * Samples use the EL saved in their source packet. A branch sample + * cannot change thread, so the thread is kept in the frontend context. */ struct thread *frontend_thread; }; @@ -1397,11 +1395,10 @@ static inline int cs_etm__instr_size(struct cs_etm_queue *etmq, static inline u64 cs_etm__first_executed_instr(struct cs_etm_packet *packet) { /* - * Return 0 for packets that have no addresses so that CS_ETM_INVAL_ADDR doesn't - * appear in samples. + * Return 0 for discontinuities so that CS_ETM_INVAL_ADDR doesn't appear + * in samples. */ - if (packet->sample_type == CS_ETM_DISCONTINUITY || - packet->sample_type == CS_ETM_EXCEPTION) + if (packet->sample_type == CS_ETM_DISCONTINUITY) return 0;
return packet->start_addr; @@ -1414,6 +1411,9 @@ u64 cs_etm__last_executed_instr(const struct cs_etm_packet *packet) if (packet->sample_type == CS_ETM_DISCONTINUITY) return 0;
+ if (packet->sample_type == CS_ETM_EXCEPTION) + return packet->pref_ret_addr - packet->last_instr_size; + return packet->end_addr - packet->last_instr_size; }
@@ -1580,6 +1580,9 @@ static void cs_etm__copy_insn(struct cs_etm_queue *etmq, return; }
+ if (packet->sample_type == CS_ETM_EXCEPTION && !sample->insn_len) + return; + cs_etm__frontend_mem_access(etmq, tidq, packet, sample->ip, sample->insn_len, (void *)sample->insn); } @@ -1606,8 +1609,11 @@ static inline u64 cs_etm__resolve_sample_time(struct cs_etm_queue *etmq, return etm->latest_kernel_timestamp; }
-static bool cs_etm__packet_has_taken_branch(struct cs_etm_packet *packet) +static bool cs_etm__packet_has_branch(struct cs_etm_packet *packet) { + if (packet->sample_type == CS_ETM_EXCEPTION) + return true; + if (packet->sample_type == CS_ETM_RANGE && packet->last_instr_taken_branch) return true; @@ -1625,7 +1631,7 @@ static void cs_etm__add_stack_event(struct cs_etm_queue *etmq, if (!etm->synth_opts.branches && !etm->synth_opts.instructions) return;
- if (!cs_etm__packet_has_taken_branch(tidq->prev_packet)) + if (!cs_etm__packet_has_branch(tidq->prev_packet)) return;
if (etmq->etm->use_thread_stack) { @@ -1737,7 +1743,8 @@ static int cs_etm__synth_last_instruction_sample(struct cs_etm_queue *etmq, !etmq->etm->synth_opts.instructions) return 0;
- if (packet->sample_type != CS_ETM_RANGE) + /* Only nonempty ranges provide a final instruction to sample. */ + if (packet->sample_type != CS_ETM_RANGE || !packet->instr_count) return 0;
ret = cs_etm__synth_instruction_sample(etmq, tidq, packet, @@ -1771,7 +1778,7 @@ static int cs_etm__synth_branch_sample(struct cs_etm_queue *etmq, if (!etm->synth_opts.branches) return 0;
- if (!cs_etm__packet_has_taken_branch(tidq->prev_packet) && + if (!cs_etm__packet_has_branch(tidq->prev_packet) && !(tidq->prev_packet->flags & (PERF_IP_FLAG_TRACE_BEGIN | PERF_IP_FLAG_TRACE_END))) return 0; @@ -1802,6 +1809,9 @@ static int cs_etm__synth_branch_sample(struct cs_etm_queue *etmq, sample.flags = tidq->prev_packet->flags; sample.cpumode = event->sample.header.misc;
+ if (tidq->prev_packet->sample_type == CS_ETM_EXCEPTION) + sample.ret_addr = tidq->prev_packet->pref_ret_addr; + cs_etm__copy_insn(etmq, tidq, tidq->prev_packet, &sample);
/* @@ -2052,21 +2062,23 @@ static int cs_etm__context(struct cs_etm_queue *etmq, return ret; }
-static int cs_etm__exception(struct cs_etm_traceid_queue *tidq) +static int cs_etm__exception(struct cs_etm_queue *etmq, + struct cs_etm_traceid_queue *tidq) { + struct cs_etm_packet *packet = tidq->packet; + /* - * When the exception packet is inserted, whether the last instruction - * in previous range packet is taken branch or not, we need to force - * to set 'prev_packet->last_instr_taken_branch' to true. This ensures - * to generate branch sample for the instruction range before the - * exception is trapped to kernel or before the exception returning. - * - * The exception packet does not describe an instruction range, so don't - * swap PACKET with PREV_PACKET. This keeps PREV_PACKET to be useful - * for generating instruction and branch samples. + * Resolve the preceding branch without adding instructions, then keep + * this exception as prev_packet until its destination is known. */ - if (tidq->prev_packet->sample_type == CS_ETM_RANGE) + if (packet->start_addr != CS_ETM_INVAL_ADDR) + return cs_etm__sample(etmq, tidq); + + /* Fall back to attributing the exception to the preceding range. */ + if (tidq->prev_packet->sample_type == CS_ETM_RANGE) { + tidq->prev_packet->flags = packet->flags; tidq->prev_packet->last_instr_taken_branch = true; + }
return 0; } @@ -2463,7 +2475,8 @@ static int cs_etm__set_sample_flags(struct cs_etm_queue *etmq, * instruction packet, set flag PERF_IP_FLAG_TRACE_END * for previous packet. */ - if (prev_packet->sample_type == CS_ETM_RANGE) + if (prev_packet->sample_type == CS_ETM_RANGE || + prev_packet->sample_type == CS_ETM_EXCEPTION) prev_packet->flags |= PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_TRACE_END; break; @@ -2495,15 +2508,23 @@ static int cs_etm__set_sample_flags(struct cs_etm_queue *etmq, PERF_IP_FLAG_CALL | PERF_IP_FLAG_INTERRUPT;
- /* - * When the exception packet is inserted, since exception - * packet is not used standalone for generating samples - * and it's affiliation to the previous instruction range - * packet; so set previous range packet flags to tell perf - * it is an exception taken branch. - */ - if (prev_packet->sample_type == CS_ETM_RANGE) - prev_packet->flags = packet->flags; + if (packet->start_addr == CS_ETM_INVAL_ADDR) + break; + + /* Resolve the preceding trace start or exception return. */ + if (prev_packet->sample_type == CS_ETM_DISCONTINUITY) + prev_packet->flags |= PERF_IP_FLAG_BRANCH | + PERF_IP_FLAG_TRACE_BEGIN; + + if (prev_packet->flags == (PERF_IP_FLAG_BRANCH | + PERF_IP_FLAG_RETURN | + PERF_IP_FLAG_INTERRUPT) && + cs_etm__is_svc_instr(etmq, tidq, packet, + packet->start_addr)) { + prev_packet->flags = PERF_IP_FLAG_BRANCH | + PERF_IP_FLAG_RETURN | + PERF_IP_FLAG_SYSCALLRET; + } break; case CS_ETM_EXCEPTION_RET: /* @@ -2531,10 +2552,12 @@ static int cs_etm__set_sample_flags(struct cs_etm_queue *etmq, * system call instruction and then calibrate the sample flag * as needed. */ - if (prev_packet->sample_type == CS_ETM_RANGE) + if (prev_packet->sample_type == CS_ETM_RANGE) { prev_packet->flags = PERF_IP_FLAG_BRANCH | PERF_IP_FLAG_RETURN | PERF_IP_FLAG_INTERRUPT; + prev_packet->last_instr_taken_branch = true; + } break; case CS_ETM_CONTEXT: case CS_ETM_EMPTY: @@ -2610,7 +2633,9 @@ static int cs_etm__process_traceid_queue(struct cs_etm_queue *etmq, * range, generate instruction sequence * events. */ - cs_etm__sample(etmq, tidq); + ret = cs_etm__sample(etmq, tidq); + if (ret) + goto out; break; case CS_ETM_CONTEXT: /* @@ -2624,13 +2649,12 @@ static int cs_etm__process_traceid_queue(struct cs_etm_queue *etmq, goto out; break; case CS_ETM_EXCEPTION: + ret = cs_etm__exception(etmq, tidq); + if (ret) + goto out; + break; case CS_ETM_EXCEPTION_RET: - /* - * If the exception packet is coming, - * make sure the previous instruction - * range packet to be handled properly. - */ - cs_etm__exception(tidq); + /* The return annotates the preceding instruction range. */ break; case CS_ETM_DISCONTINUITY: /* diff --git a/tools/perf/util/cs-etm.h b/tools/perf/util/cs-etm.h index e40b9b7d0a9bd90f089c2f20527332264ead4249..bf0e0f5a9711f8d450ea8a0f9d67aa5e8d7204ab 100644 --- a/tools/perf/util/cs-etm.h +++ b/tools/perf/util/cs-etm.h @@ -173,6 +173,7 @@ struct cs_etm_queue; struct cs_etm_packet { enum cs_etm_sample_type sample_type; enum cs_etm_isa isa; + /* For exceptions: source PC, or CS_ETM_INVAL_ADDR. */ u64 start_addr; union { u64 end_addr; /* Range packets */
Add an AArch64 branch_not_taken_loop workload and a CoreSight shell test checking IRQ entry and exit PCs. Use cpu-clock to generate timer interrupts.
Retry at different timer frequencies and fail if no complete pair is captured after all three attempts.
For example, an expected hw int/iret pair is:
hw int 5cc928 => ffff800080010c80 insn: 63 04 00 f1 ... iret ffff800080012284 => 5cc928 insn: e0 03 9f d6
The entry PC is the architectural resume address, here SUBS X3, X3, #1 at 0x5cc928. After the IRQ is handled, ERET returns to the same PC to continue execution.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/tests/builtin-test.c | 1 + tools/perf/tests/shell/coresight/irq_entry_exit.sh | 36 +++++ tools/perf/tests/shell/lib/coresight_exception.sh | 164 +++++++++++++++++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 + tools/perf/tests/workloads/branch_not_taken_loop.c | 33 +++++ 6 files changed, 237 insertions(+)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c index 8dd72cd97cdb2de0f80f108cd72b9264c073e2a0..24e095d4d20cee6a6aae7cfda364fb237ef74b30 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -191,6 +191,7 @@ static struct test_workload *workloads[] = { &workload__datasym, &workload__landlock, &workload__traploop, + &workload__branch_not_taken_loop, &workload__inlineloop, &workload__jitdump, &workload__context_switch_loop, diff --git a/tools/perf/tests/shell/coresight/irq_entry_exit.sh b/tools/perf/tests/shell/coresight/irq_entry_exit.sh new file mode 100755 index 0000000000000000000000000000000000000000..135483b8d3532b26610dcabe7fab91f025ee0cb5 --- /dev/null +++ b/tools/perf/tests/shell/coresight/irq_entry_exit.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# CoreSight IRQ entry and exit (exclusive) + +. "$(dirname "$0")/../lib/coresight_exception.sh" + +test_irq_entry_exit() +{ + # IRQs must return to the interrupted PC. Retry at different timer + # frequencies if trace gaps leave no complete entry/exit pair. + for freq in 1000 4000 10000; do + echo "Recording timer IRQs at $freq Hz" + if ! record_trace_with_cpu_clock "$freq" branch_not_taken_loop; then + cat "$tmpdir/record.log" + echo "Failed to record the IRQ workload" + return 1 + fi + + decode_trace || return 1 + + check_exception_pairs "hw int" "iret" 0 + result=$? + case $result in + 0) return 0 ;; + 1) cat "$tmpdir/script.log"; return 1 ;; + 2) echo "No complete IRQ pair in this recording" ;; + *) return "$result" ;; + esac + done + + echo "FAIL: No complete IRQ pair after three recordings" + return 1 +} + +setup_exception_test || exit $? +test_irq_entry_exit diff --git a/tools/perf/tests/shell/lib/coresight_exception.sh b/tools/perf/tests/shell/lib/coresight_exception.sh new file mode 100644 index 0000000000000000000000000000000000000000..2d5a24ce3b5494e69973f61f4372e432c803952e --- /dev/null +++ b/tools/perf/tests/shell/lib/coresight_exception.sh @@ -0,0 +1,164 @@ +# SPDX-License-Identifier: GPL-2.0 +# Common helpers for CoreSight exception entry and return tests. + +cleanup() +{ + rm -rf "$tmpdir" +} + +record_trace_with_cpu_clock() +{ + local cf="$tmpdir/ctl" + local af="$tmpdir/ack" + local clock_opts=() + + record_freq=$1 + shift + + # A zero frequency omits the cpu-clock event. + if [ "$record_freq" -gt 0 ]; then + # Generate timer IRQs without delivering a signal to the workload. + clock_opts=(-e cpu-clock:u -F "$record_freq") + fi + + # FIFO control bounds the trace to the workload. + # --kcore supplies the running kernel's instructions for decoding. + taskset -c "$cpu" perf record -B --no-bpf-event --per-thread --kcore \ + -e cs_etm/timestamp=0/uk "${clock_opts[@]}" \ + -m,4M -D -1 --control "fifo:$cf,$af" \ + -o "$tmpdir/data" -- \ + perf test --record-ctl "fifo:$cf,$af" -w "$@" \ + > "$tmpdir/record.log" 2>&1 +} + +record_trace() +{ + record_trace_with_cpu_clock 0 "$@" +} + +decode_trace() +{ + local sw_fields=() + + # perf script rejects -F sw: when no software event was recorded. + if [ "$record_freq" -gt 0 ]; then + sw_fields=(-F sw:) + fi + + if ! perf script -i "$tmpdir/data" --itrace=b \ + -F hw:ip,addr,flags,insn "${sw_fields[@]}" \ + > "$tmpdir/script" 2> "$tmpdir/script.log"; then + cat "$tmpdir/script.log" + echo "Failed to decode the exception trace" + return 1 + fi +} + +# Arguments: entry pattern, exit pattern, expected return PC - entry PC. +# Return 2 if no complete pair survives the trace gaps. +check_exception_pairs() +{ + local difference=$3 entry_pc resume_pc pairs=0 + + awk -v entry_pattern="$1" -v exit_pattern="$2" ' + function address(pc) { + sub(/^0[xX]/, "", pc) + sub(/^0+/, "", pc) + return "0x" (pc == "" ? "0" : tolower(pc)) + } + BEGIN { + entry_pattern = "^[[:space:]]*(" entry_pattern ")([[:space:]]|$)" + exit_pattern = "^[[:space:]]*(" exit_pattern ")([[:space:]]|$)" + } + NF { + # A trace boundary invalidates all pending entries, including nested ones. + if ($1 == "tr" && ($2 == "strt" || $2 == "end")) { + depth = 0 + next + } + + is_entry = ($1 == "hw" && $2 == "int") || $1 == "int" || $1 == "syscall" + is_exit = $1 == "iret" || $1 == "sysret" + if (!is_entry && !is_exit) + next + + # Branch fields are: flags source => destination [insn: bytes]. + arrow = 0 + for (i = 1; i <= NF; i++) { + if ($i == "=>") { + arrow = i + break + } + } + if (arrow < 3 || arrow == NF) { + print "FAIL: Unexpected perf script output: " $0 > "/dev/stderr" + errors++ + depth = 0 + next + } + + # Track every exception so an unrelated nested IRET cannot close + # a selected outer entry. Match the requested flags at each depth. + if (is_entry) { + entry_pc[++depth] = address($(arrow - 1)) + selected[depth] = $0 ~ entry_pattern + next + } + if (!depth) + next + + if (selected[depth] && $0 ~ exit_pattern) + print entry_pc[depth], address($(arrow + 1)) + depth-- + } + END { + if (errors) + exit 1 + }' "$tmpdir/script" > "$tmpdir/exception_pairs" || return 1 + + # Bash uses integer arithmetic, preserving all bits of 64-bit PCs. + while read -r entry_pc resume_pc; do + if [ "$((resume_pc - entry_pc))" -ne "$difference" ]; then + printf "FAIL: Entry %s returns to %s, expected PC difference %d\n" \ + "$entry_pc" "$resume_pc" "$difference" + return 1 + fi + pairs=$((pairs + 1)) + done < "$tmpdir/exception_pairs" + + if [ "$pairs" -eq 0 ]; then + echo "No complete exception entry/exit pair" + return 2 + fi + printf "Checked %d exception pairs with PC difference %d\n" "$pairs" "$difference" +} + +setup_exception_test() +{ + [ "$(uname -m)" = aarch64 ] || return 2 + perf check feature -q libopencsd || return 2 + [ -d /sys/bus/event_source/devices/cs_etm ] || return 2 + + export LC_ALL=C + command -v taskset >/dev/null 2>&1 || return 2 + + tmpdir=$(mktemp -d /tmp/perf-cs-etm-exception.XXXXXX) || return 1 + trap cleanup EXIT + trap 'exit 1' HUP INT TERM + mkfifo "$tmpdir/ctl" "$tmpdir/ack" || return 1 + + # Keep each exception entry and return in the same trace queue. + cpu=$(awk '/Cpus_allowed_list:/ { + split($2, cpus, /[-,]/) + print cpus[1] + }' /proc/self/status) + + # Check permissions, FIFO control and the trace sink before testing. + if ! record_trace callchain; then + cat "$tmpdir/record.log" + echo "[Skip] Cannot record user/kernel CoreSight trace with --kcore" + return 2 + fi + + return 0 +} diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h index 59b66fe34a71420715ca8e04bf5f0f181b2c9cf9..989208fcd417a676d03e2372e8a71ecf2059b9c6 100644 --- a/tools/perf/tests/tests.h +++ b/tools/perf/tests/tests.h @@ -264,6 +264,7 @@ DECLARE_WORKLOAD(brstack); DECLARE_WORKLOAD(datasym); DECLARE_WORKLOAD(landlock); DECLARE_WORKLOAD(traploop); +DECLARE_WORKLOAD(branch_not_taken_loop); DECLARE_WORKLOAD(inlineloop); DECLARE_WORKLOAD(jitdump); DECLARE_WORKLOAD(context_switch_loop); diff --git a/tools/perf/tests/workloads/Build b/tools/perf/tests/workloads/Build index 18fceddccb56c01388383ee345d38df1acff8395..d3030bd9c0752a5bc8743a4539a0ac365d215719 100644 --- a/tools/perf/tests/workloads/Build +++ b/tools/perf/tests/workloads/Build @@ -9,6 +9,7 @@ perf-test-y += brstack.o perf-test-y += datasym.o perf-test-y += landlock.o perf-test-y += traploop.o +perf-test-y += branch_not_taken_loop.o perf-test-y += inlineloop.o perf-test-y += jitdump.o perf-test-y += context_switch_loop.o @@ -26,6 +27,7 @@ CFLAGS_leafloop.o = -g -O0 -fno-inline -fno-omit-frame-pointer -U_FORTIF CFLAGS_brstack.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_datasym.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_traploop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE +CFLAGS_branch_not_taken_loop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_inlineloop.o = -g -O2 CFLAGS_deterministic.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_named_threads.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE diff --git a/tools/perf/tests/workloads/branch_not_taken_loop.c b/tools/perf/tests/workloads/branch_not_taken_loop.c new file mode 100644 index 0000000000000000000000000000000000000000..d5e1d6119e143c5a89bff88861ff4d03e84750ec --- /dev/null +++ b/tools/perf/tests/workloads/branch_not_taken_loop.c @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/compiler.h> +#include "../tests.h" + +#ifdef __aarch64__ +static void branch_not_taken(void) +{ + /* Keep x1 > x2 so B.LS is never taken. */ + asm volatile( + "mov x1, #2\n" + "mov x2, #1\n" + "movz x3, #0xffff\n" + "movk x3, #0x0010, lsl #16\n" + "1: cmp x1, x2\n" + ".Ltest_branch:\n" + "b.ls 2f\n" + ".Lfallthrough:\n" + "subs x3, x3, #1\n" + "b.ne 1b\n" + "2:\n" + : : : "x1", "x2", "x3", "cc"); +} +#else +static void branch_not_taken(void) { } +#endif + +static int branch_not_taken_loop(int argc __maybe_unused, const char **argv __maybe_unused) +{ + branch_not_taken(); + return 0; +} + +DEFINE_WORKLOAD(branch_not_taken_loop);
Record the existing callchain workload, which issues gettid() through SVC. The test requires syscall/sysret pairs to advance the PC by four bytes.
For example, an expected syscall/sysret pair is:
syscall bb76e4 => ffff800080010c00 insn: 01 00 00 d4 ... sysret ffff800080012284 => bb76e8 insn: e0 03 9f d6
The entry instruction is SVC #0 at 0xbb76e4. On syscall completion, ERET returns to 0xbb76e8, the instruction immediately after SVC.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- .../tests/shell/coresight/syscall_entry_exit.sh | 32 ++++++++++++++++++++++ 1 file changed, 32 insertions(+)
diff --git a/tools/perf/tests/shell/coresight/syscall_entry_exit.sh b/tools/perf/tests/shell/coresight/syscall_entry_exit.sh new file mode 100755 index 0000000000000000000000000000000000000000..cf49f9961bf710da2738ef92c12887d5fc1f06d1 --- /dev/null +++ b/tools/perf/tests/shell/coresight/syscall_entry_exit.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# CoreSight syscall entry and exit (exclusive) + +. "$(dirname "$0")/../lib/coresight_exception.sh" + +test_syscall_entry_exit() +{ + local attempt result + + # callchain() invokes gettid() through SVC, which returns to PC + 4. + for attempt in 1 2 3; do + echo "Recording syscalls (attempt $attempt/3)" + if ! record_trace callchain; then + cat "$tmpdir/record.log" + echo "Failed to record the callchain workload" + return 1 + fi + decode_trace || return 1 + check_exception_pairs "syscall" "sysret" 4 + result=$? + if [ "$result" -ne 2 ]; then + return "$result" + fi + done + + echo "FAIL: No complete syscall pair after three recordings" + return 1 +} + +setup_exception_test || exit $? +test_syscall_entry_exit
Add page_fault_loop to repeatedly discard an anonymous page with MADV_DONTNEED and read it again. Each demand fault is resolved without signal delivery, so the load must be retried at the same PC.
For example, an expected int/iret pair is:
int 5ccadc => ffff800080010c00 insn: 00 00 40 39 ... iret ffff800080012284 => 5ccadc insn: e0 03 9f d6
The LDRB W0, [X0] at 0x5ccadc faults before completing. After resolving the page fault, the kernel returns via ERET to the same PC to retry it.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- tools/perf/tests/builtin-test.c | 1 + .../perf/tests/shell/coresight/abort_entry_exit.sh | 32 +++++++++++++++++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/workloads/Build | 2 ++ tools/perf/tests/workloads/page_fault_loop.c | 37 ++++++++++++++++++++++ 5 files changed, 73 insertions(+)
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c index 24e095d4d20cee6a6aae7cfda364fb237ef74b30..86698b86657f45b094755583800062c37e16ca8b 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -192,6 +192,7 @@ static struct test_workload *workloads[] = { &workload__landlock, &workload__traploop, &workload__branch_not_taken_loop, + &workload__page_fault_loop, &workload__inlineloop, &workload__jitdump, &workload__context_switch_loop, diff --git a/tools/perf/tests/shell/coresight/abort_entry_exit.sh b/tools/perf/tests/shell/coresight/abort_entry_exit.sh new file mode 100755 index 0000000000000000000000000000000000000000..98aeb2b1068e9847d87e3b25aeabf6da114c6ae2 --- /dev/null +++ b/tools/perf/tests/shell/coresight/abort_entry_exit.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# CoreSight abort entry and exit (exclusive) + +. "$(dirname "$0")/../lib/coresight_exception.sh" + +test_abort_entry_exit() +{ + local attempt result + + # A demand page fault retries the load at the same PC after resolution. + for attempt in 1 2 3; do + echo "Recording page faults (attempt $attempt/3)" + if ! record_trace page_fault_loop; then + cat "$tmpdir/record.log" + echo "Failed to record the page fault workload" + return 1 + fi + decode_trace || return 1 + check_exception_pairs "int" "iret" 0 + result=$? + if [ "$result" -ne 2 ]; then + return "$result" + fi + done + + echo "FAIL: No complete abort pair after three recordings" + return 1 +} + +setup_exception_test || exit $? +test_abort_entry_exit diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h index 989208fcd417a676d03e2372e8a71ecf2059b9c6..8c399840a735910d4344eba928075e3ec6b1f81c 100644 --- a/tools/perf/tests/tests.h +++ b/tools/perf/tests/tests.h @@ -265,6 +265,7 @@ DECLARE_WORKLOAD(datasym); DECLARE_WORKLOAD(landlock); DECLARE_WORKLOAD(traploop); DECLARE_WORKLOAD(branch_not_taken_loop); +DECLARE_WORKLOAD(page_fault_loop); DECLARE_WORKLOAD(inlineloop); DECLARE_WORKLOAD(jitdump); DECLARE_WORKLOAD(context_switch_loop); diff --git a/tools/perf/tests/workloads/Build b/tools/perf/tests/workloads/Build index d3030bd9c0752a5bc8743a4539a0ac365d215719..43cc4da6fb0786de8bb5c52717c2ad5fdbaa639c 100644 --- a/tools/perf/tests/workloads/Build +++ b/tools/perf/tests/workloads/Build @@ -10,6 +10,7 @@ perf-test-y += datasym.o perf-test-y += landlock.o perf-test-y += traploop.o perf-test-y += branch_not_taken_loop.o +perf-test-y += page_fault_loop.o perf-test-y += inlineloop.o perf-test-y += jitdump.o perf-test-y += context_switch_loop.o @@ -28,6 +29,7 @@ CFLAGS_brstack.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_datasym.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_traploop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_branch_not_taken_loop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE +CFLAGS_page_fault_loop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_inlineloop.o = -g -O2 CFLAGS_deterministic.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE CFLAGS_named_threads.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE diff --git a/tools/perf/tests/workloads/page_fault_loop.c b/tools/perf/tests/workloads/page_fault_loop.c new file mode 100644 index 0000000000000000000000000000000000000000..4afbc4ab0a55400360facdc3a99ea5a5a9d52596 --- /dev/null +++ b/tools/perf/tests/workloads/page_fault_loop.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/compiler.h> +#include <stdio.h> +#include <sys/mman.h> +#include <unistd.h> +#include "../tests.h" + +static int page_fault_loop(int argc __maybe_unused, const char **argv __maybe_unused) +{ + size_t size = getpagesize(); + char *page; + int ret = 1; + + page = mmap(NULL, size, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (page == MAP_FAILED) { + perror("mmap"); + return 1; + } + + for (int i = 0; i < 256; i++) { + /* Drop the PTE so the kernel must resolve a fault on each read. */ + if (madvise(page, size, MADV_DONTNEED)) { + perror("madvise"); + goto out; + } + if (READ_ONCE(*page)) { + fprintf(stderr, "Anonymous page is not zero-filled\n"); + goto out; + } + } + ret = 0; +out: + munmap(page, size); + return ret; +} + +DEFINE_WORKLOAD(page_fault_loop);
Use traploop's EL0 read of ID_AA64ISAR0_EL1 to check an emulated instruction. The kernel emulates MRS and advances the return PC by four bytes.
Select MRS entries by instruction encoding and reuse the common FIFO recorder and pair checker. An instruction page fault at MRS also matches the opcode filter and returns to the same PC. Retry recordings when the pair check fails or finds no complete pair, and fail after three unsuccessful attempts. Recording and decoding errors fail immediately.
For example, an expected int/iret pair is:
int 5cc808 => ffff800080010c00 insn: 00 06 38 d5 ... iret ffff800080012284 => 5cc80c insn: e0 03 9f d6
The entry instruction is MRS X0, ID_AA64ISAR0_EL1. After emulating the read at 0x5cc808, the kernel returns via ERET to the next instruction at 0x5cc80c.
Reviewed-by: James Clark james.clark@linaro.org Signed-off-by: Leo Yan leo.yan@arm.com --- .../perf/tests/shell/coresight/trap_entry_exit.sh | 39 ++++++++++++++++++++++ 1 file changed, 39 insertions(+)
diff --git a/tools/perf/tests/shell/coresight/trap_entry_exit.sh b/tools/perf/tests/shell/coresight/trap_entry_exit.sh new file mode 100755 index 0000000000000000000000000000000000000000..4521e4f4ba58100c7da7a915d729df7ad814f031 --- /dev/null +++ b/tools/perf/tests/shell/coresight/trap_entry_exit.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# CoreSight emulated instruction entry and exit (exclusive) + +. "$(dirname "$0")/../lib/coresight_exception.sh" + +test_trap_entry_exit() +{ + local attempt result + + # traploop reads ID_AA64ISAR0_EL1 from EL0. The kernel emulates MRS and + # advances the saved PC, so IRET must resume four bytes after the trap. + for attempt in 1 2 3; do + echo "Recording emulated instructions (attempt $attempt/3)" + if ! record_trace traploop 256; then + cat "$tmpdir/record.log" + echo "Failed to record the traploop workload" + return 1 + fi + decode_trace || return 1 + # Select MRS Xt, ID_AA64ISAR0_EL1; the low five bits select Xt. + check_exception_pairs "int.*insn: [01][[:xdigit:]] 06 38 d5" "iret" 4 + result=$? + + # An instruction page fault at MRS can also match the opcode filter. + # Retry failed checks to tolerate such transient faults. + case $result in + 0) return 0 ;; + 1|2) continue ;; + *) return "$result" ;; + esac + done + + echo "FAIL: No valid trap pair after three recordings" + return 1 +} + +setup_exception_test || exit $? +test_trap_entry_exit
On 30/10/2026 15:13, Leo Yan wrote:
CoreSight currently models exception entry by changing the preceding instruction range into a taken branch. This can give an IRQ the source PC of an instruction that already retired and overwrite a real branch immediately before the exception.
For an untaken B.LS followed by an IRQ, perf script currently reports:
hw int 4000f8 => ffff800080010c80 b.ls #0x400118 ... iret ffff800080012284 => 4000fc eret
The hardware trace supplies 0x4000fc as the preferred return address. For an IRQ, this is the architectural resume PC (Arm ARM, R_VBQMV). B.LS has retired, and the saved PC identifies the boundary before the following MOV. Using this PC as the IRQ source reflects the architectural state at exception entry:
hw int 4000fc => ffff800080010c80 movz x2, #0x1796 ... iret ffff800080012284 => 4000fc eret
The IRQ sample represents the transfer from this architectural execution position to the handler. The series synthesizes exception entries from exception packets, preserving the preceding branch and its outcome.
The supporting changes:
- Let decoders supply sample.ret_addr so later instruction fetching cannot change thread-stack return addresses. Apply this to Intel PT asynchronous samples as well.
- Prepare packet ISA and instruction-size handling and share sample synthesis helpers. Mark untaken branches and break history when instruction memory is unavailable.
- Add a thread-stack regression test and four AArch64 CoreSight tests. IRQs and page faults must resume at the entry PC; SVC and emulated MRS must resume four bytes later.
Based on the AI search and test on my x86 machine, this matches perf's Intel PT handling of IRQs. Intel PT records the next instruction's IP in the FUP packet, and Perf uses that IP as the interrupt sample's source.
This series is verified on Orion6 board with "perf test coresight".
Signed-off-by: Leo Yan leo.yan@arm.com
Changes in v3:
- Introduced the pref_ret_addr/end_addr union (James).
- Clarified zero-size A32/A64 exception handling and why T32 retains per-address instruction decoding (James).
- Retreed missing exception pairs for up to three recordings, then fail instead of skipping (James).
- Shorten the IRQ workload (James).
- Fix FIFO argument quoting and remove redundant ShellCheck source directives (James).
- Checked instruction length before copying bytes into sample->insn (Sashiko).
- Retried MRS checks when instruction page faults may match the opcode filter (Sashiko).
- Rebased onto the updated perf tree and collect review tags.
- Link to v2: https://lore.kernel.org/r/20260923-perf_cs_etm_fix_non_taken-v2-0-6ab8c07a54...
Changes in v2:
- Rework the fix around exception packets to preserve both exception entries and preceding branches.
- Add explicit return addresses and the Intel PT asynchronous-branch fix.
- Split out packet/synthesis preparation, record not-taken branches and handle unreadable instruction memory.
- Add thread-stack regression coverage and four CoreSight tests.
- Link to v1: https://lore.kernel.org/r/20260713-perf_cs_etm_fix_non_taken-v1-0-4561607fc6...
Leo Yan (14): perf sample: Allow decoders to supply branch return addresses perf intel-pt: Preserve return addresses for asynchronous branches perf cs-etm: Break branch history when instruction memory is unavailable perf cs-etm: Centralize packet ISA initialization perf cs-etm: Use the recorded instruction size for A32 and A64 perf cs-etm: Mark branches that were not taken perf cs-etm: Factor out final instruction sample synthesis perf cs-etm: Centralize branch sample synthesis checks perf cs-etm: Classify exception calls using the exception packet perf cs-etm: Synthesize exception entries separately from branches perf tests: Check CoreSight IRQ entry and exit perf tests: Check CoreSight syscall entry and exit perf tests: Check CoreSight abort entry and exit perf tests: Check CoreSight emulated instruction entry and exit
tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 3 + .../perf/tests/shell/coresight/abort_entry_exit.sh | 32 +++ tools/perf/tests/shell/coresight/irq_entry_exit.sh | 36 +++ .../tests/shell/coresight/syscall_entry_exit.sh | 32 +++ .../perf/tests/shell/coresight/trap_entry_exit.sh | 39 ++++ tools/perf/tests/shell/lib/coresight_exception.sh | 164 ++++++++++++++ tools/perf/tests/tests.h | 3 + tools/perf/tests/thread-stack.c | 106 +++++++++ tools/perf/tests/workloads/Build | 4 + tools/perf/tests/workloads/branch_not_taken_loop.c | 33 +++ tools/perf/tests/workloads/page_fault_loop.c | 37 ++++ tools/perf/util/cs-etm-decoder/cs-etm-decoder.c | 106 ++++++--- tools/perf/util/cs-etm.c | 246 +++++++++++---------- tools/perf/util/cs-etm.h | 6 +- tools/perf/util/intel-pt.c | 7 + tools/perf/util/sample.c | 1 + tools/perf/util/sample.h | 5 + tools/perf/util/thread-stack.c | 5 +- 19 files changed, 714 insertions(+), 152 deletions(-)
base-commit: 4626d370ae89b8af4d46a45372b269f926ec662d change-id: 20260713-perf_cs_etm_fix_non_taken-5b4d7f73f41e
Best regards,
Reviewed-by: James Clark james.clark@linaro.org
-
James Clark -
Leo Yan