Trace-cmd is a tool implemented by Steven Rostedt. Its purpose is to facilitate the users’ control over the kernel tracing mechanism and to add support for more elaborate parsing and display of the trace output through plugins.

The trace-cmd tool can be found in the following repository:


To download and install it, do:

$ git clone git://
$ cd trace-cmd
$ make && sudo make install

The above command will build and install trace-cmd tool in /usr/local/bin. If you prefer to install it in another directory, do instead:

$ make && make prefix=<dir> install

Using the trace-cmd tool, the debug filesystem will be automatically mounted (if it is not already mounted) on /sys/kernel/debug/.

Trace-cmd writes the generated traces in a file, called trace.dat, and displays the content of this file either with the default format, as defined in the corresponding trace event’s definition, or it uses a plugin, if available, to parse and display the file content.

Trace-cmd plugins are written in C or in Python and can be found under the directory /usr/local/lib/trace-cmd/plugins or <dir>/lib/trace-cmd/plugins.

For the Python plugins to work, you need to install python-dev and swig packages.

As it was described in the previous post, to enable a trace event via direct access to the debugfs tracing files we do:

$ echo 1 > events/<trace system>/<trace event>/enable
$ echo <trace system>:<trace event> >> set_event

And to disable it:

$ echo 0 > events/<trace system>/<trace event>/enable
$ echo '!<trace system>:<trace event>' >> set_event

To view the trace output, we do:

$ cat trace

An alternative way, using trace-cmd, would be:

$ trace-cmd record -e <trace system>:<trace event>
$ trace-cmd report

‘record’ will enable the trace event and record its trace in the trace.dat file until Ctrl-C is pressed, while ‘report’ will load the plugin if available and will display the trace.
A subsequent record command will overwrite the previous contents in trace.dat.

In case you want to record more than one trace events, for instance:
– all the trace events for the xhci-hcd, you can do:

$ trace-cmd record -e xhci-hcd

– multiple trace events, you can do:

$ trace-cmd record -e xhci-hcd:xhci_dbg_context_change -e xhci-hcd:xhci_cmd_completion

And then filter the report according to a specific trace event:

$ trace-cmd report -F xhci_cmd_completion

Or even filter according to the value of an event’s structure field:

$ trace-cmd report -F xhci_cmd_completion:dma==0x374060c0

The general format used for filtering is:

$ trace-cmd report -F xhci-hcd/xhci_cmd_completion:dma==0x374060c0

The ‘report’ command will use the plugin, if available, to report the traces. In case you don’t want to load any plugins and you want to view the traces with the default format (not the raw trace) do:

$ trace-cmd report -N

If you are a raw foodist do:

$ trace-cmd report -R

To list all available trace events do:

$ trace-cmd list -e

To list all available tracers do:

$ trace-cmd list -t

To list all available plugins do:

$ trace-cmd list -P

This command, though, does not list my plugin written in python. So maybe it lists only the C plugins under ../lib/trace-cmd/plugins.

When finished with tracing and you want back your performance and buffers do:

$ trace-cmd reset

I think the most important commands (to me) have been covered so far. An extended documentation on the trace-cmd commands and options can be found in the downloaded trace-cmd files under Documentation/ subdirectory.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s