Trace-cmd is a tool implemented by Steven Rostedt. Its purpose is to facilitate the users’ control over the kernel tracing mechanism and to add support for more elaborate parsing and display of the trace output through plugins.
The trace-cmd tool can be found in the following repository:
To download and install it, do:
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git $ cd trace-cmd $ make && sudo make install
The above command will build and install trace-cmd tool in /usr/local/bin. If you prefer to install it in another directory, do instead:
$ make && make prefix=<dir> install
Using the trace-cmd tool, the debug filesystem will be automatically mounted (if it is not already mounted) on /sys/kernel/debug/.
Trace-cmd writes the generated traces in a file, called trace.dat, and displays the content of this file either with the default format, as defined in the corresponding trace event’s definition, or it uses a plugin, if available, to parse and display the file content.
Trace-cmd plugins are written in C or in Python and can be found under the directory /usr/local/lib/trace-cmd/plugins or <dir>/lib/trace-cmd/plugins.
For the Python plugins to work, you need to install python-dev and swig packages.
As it was described in the previous post, to enable a trace event via direct access to the debugfs tracing files we do:
$ echo 1 > events/<trace system>/<trace event>/enable OR $ echo <trace system>:<trace event> >> set_event
And to disable it:
$ echo 0 > events/<trace system>/<trace event>/enable OR $ echo '!<trace system>:<trace event>' >> set_event
To view the trace output, we do:
$ cat trace
An alternative way, using trace-cmd, would be:
$ trace-cmd record -e <trace system>:<trace event> $ trace-cmd report
‘record’ will enable the trace event and record its trace in the trace.dat file until Ctrl-C is pressed, while ‘report’ will load the plugin if available and will display the trace.
A subsequent record command will overwrite the previous contents in trace.dat.
In case you want to record more than one trace events, for instance:
– all the trace events for the xhci-hcd, you can do:
$ trace-cmd record -e xhci-hcd
– multiple trace events, you can do:
$ trace-cmd record -e xhci-hcd:xhci_dbg_context_change -e xhci-hcd:xhci_cmd_completion
And then filter the report according to a specific trace event:
$ trace-cmd report -F xhci_cmd_completion
Or even filter according to the value of an event’s structure field:
$ trace-cmd report -F xhci_cmd_completion:dma==0x374060c0
The general format used for filtering is:
$ trace-cmd report -F xhci-hcd/xhci_cmd_completion:dma==0x374060c0
The ‘report’ command will use the plugin, if available, to report the traces. In case you don’t want to load any plugins and you want to view the traces with the default format (not the raw trace) do:
$ trace-cmd report -N
If you are a raw foodist do:
$ trace-cmd report -R
To list all available trace events do:
$ trace-cmd list -e
To list all available tracers do:
$ trace-cmd list -t
To list all available plugins do:
$ trace-cmd list -P
This command, though, does not list my plugin written in python. So maybe it lists only the C plugins under ../lib/trace-cmd/plugins.
When finished with tracing and you want back your performance and buffers do:
$ trace-cmd reset
I think the most important commands (to me) have been covered so far. An extended documentation on the trace-cmd commands and options can be found in the downloaded trace-cmd files under Documentation/ subdirectory.