Hi Anshuman,
On Mon, 23 Nov 2020 at 03:40, Anshuman Khandual anshuman.khandual@arm.com wrote:
Hello Mike,
On 11/16/20 8:30 PM, Mike Leach wrote:
Hi Anshuman,
I've not looked in detail at this set yet, but having skimmed through it I do have an initial question about the handling of wrapped data buffers.
With the ETR/ETB we found an issue with the way perf concatenated data captured from the hardware buffer into a single contiguous data block. The issue occurs when a wrapped buffer appears after another buffer in the data file. In a typical session perf would stop trace and copy the hardware buffer multiple times into the auxtrace buffer.
The hardware buffer and perf aux trace buffer are the same for TRBE and hence there is no actual copy involved. Trace data gets pushed into the user space via perf_aux_output_end() either via etm_event_stop() or via the IRQ handler i.e arm_trbe_irq_handler(). Data transfer to user space happens via updates to perf aux buffer indices i.e head, tail, wake up. But logically, they will appear as a stream of records to the user space while parsing perf.data file.
Understood - I suspected this would use direct write to the aux trace buffer, but the principle is the same. TRBE determines the location of data in the buffer so even without a copy, it is possible to get multiple TRBE "buffers" in the auxbuffer as the TRBE is stopped and restarted. The later copy to userspace is independent of this.
e.g.
For ETR/ETB we have a fixed length hardware data buffer - and no way of detecting buffer wraps using interrupts as the tracing is in progress.
TRBE has an interrupt. Hence there will be an opportunity to insert any additional packets if required to demarcate pre and post IRQ trace data streams.
If the buffer is not full at the point that perf transfers it then the data will look like this:-
- <async><synced trace data>
easy to decode, we can see the async at the start of the data - which would be the async issued at the start of trace.
Just curious, what makes the tracer to generate the <async> trace packet. Is there an explicit instruction or that is how the tracer starts when enabled ?
ETM / ETE will generate an async at the start of trace, and then periodically afterwards.
If the buffer wraps we see this:-
- <unsynced trace data><async><synced trace data>
Again no real issue, the decoder will skip to the async and trace from there - we lose the unsynced data.
Could you please elaborate more on the difference between sync and async trace data ?
The decoder will start reading trace from the start of the buffer. Unsynced trace is trace data that appears before the first async packet. We cannot decode this as we do not know where the packet boundaries are. Synced trace is any data after the first async packet - the async enables us to determine where the packet boundaries are so we can now determine the packets and decode the trace.
For an unwrapped buffer, we always see the first async that the ETE generated when the trace generation was started. In a wrapped buffer we search till we find an async generated as part of the periodic async packets.
Now the problem occurs when multiple transfers of data occur. We can see the following appearing as contiguous trace in the auxtrace buffer:-
- < async><synced trace data><unsynced trace data><async><synced trace data>
So there is an wrap around event between <synced trace data> and <unsynced trace data> ? Are there any other situations where this might happen ?
Not that I am aware of.
Now the decoder cannot spot the point that the synced data from the first capture ends, and the unsynced data from the second capture begins.
Got it.
This means it will continue to decode into the unsynced data - which will result in incorrect trace / outright errors. To get round this for ETR/ETB the driver will insert barrier packets into the datafile if a wrap event is detected.
But you mentioned there are on IRQs on ETR/ETB. So how the wrap event is even detected ?
A bit in the status register tells us the buffer is full - i.e. the write pointer has wrapped around to the location it started at. We cannot tell how far, or if multiple wraps have occurred, just that the event has occurred.
- <async><synced trace data><barrier><unsynced trace
data><async><synced trace data>
This <barrier> has the effect of resetting the decoder into the unsynced state so that the invalid trace is not decoded. This is a workaround we have to do to handle the limitations of the ETR / ETB trace hardware.
Got it.
For TRBE we do have interrupts, so it should be possible to prevent the buffer wrapping in most cases - but I did see in the code that there are handlers for the TRBE buffer wrap management event. Are there other factors in play that will prevent data pattern 3) from appearing in the auxtrace buffer ?
On TRBE, the buffer wrapping cannot happen without generating an IRQ. I would assume that ETE will then start again with an <async> data packet first when the handler returns.
This would only occur if the ETE was stopped and flushed prior to the wrap event. Does this happen? I am assuming that the sink is independent from the ETE, as ETM are from ETR.
Otherwise we might also have to insert a similar barrier packet for the user space tool to reset. As trace data should not get lost during an wrap event,
My understanding is that if a wrap has even occurred, then data is already lost.
ETE should complete the packet after the handler returns, hence aux buffer should still have logically contiguous stream of <synced trace data> to decode. I am not sure right now, but will look into this.
So you are relying on backpressure to stop ETE emitting packets? This could result in trace being lost due to overflow if the IRQ is not handled sufficiently quickly/.
Regards
Mike
- Anshuman
-- Mike Leach Principal Engineer, ARM Ltd. Manchester Design Centre. UK