Linux Kernel GDB tracepoint module (KGTP)

KGTP is a comprehensive dynamic tracer for analysing Linux kernel and application (including Android) problems on production systems in real time.

To use it, you don't need patch or rebuild the Linux kernel. Just build KGTP module and insmod it is OK.

It makes Linux Kernel supply a GDB remote debug interface. Then GDB in current machine or remote machine can debug and trace Linux kernel and user space program through GDB tracepoint and some other functions without stopping the Linux Kernel.

And even if the board doesn't have GDB on it and doesn't have interface for remote debug. It can debug the Linux Kernel using offline debug (See /sys/kernel/debug/gtpframe and offline debug).

Quick config and start KGTP

Get help or report issues

Table of different between GDB debug normal program and KGTP

This table is for the people that have experience using GDB debug normal program. It will help you understand and remember the function of KGTP.

Howto use GDB control KGTP trace and debug Linux kernel

Direct access the current value in normal mode

After GDB connect to KGTP, if it doesn't select any a entry of trace frame bufffer with GDB command "tfind", GDB in the normal mode. Then you can direct access the current value of memory (Linux kernel or the user space program) and the trace state variables without stop anything.

The memory of Linux kernel

Or you can access to the first entry of "static LIST_HEAD(modules)" with following command:

Or you can access to the CPU0 memory info of "DEFINE_PER_CPU(struct device *, mce_device);":

If you want show more than one variables with one GDB command, please use following example:

the trace state variables

GDB tracepoint

Tracepoint is that GDB define some addresses and some actions and put them to the target (KGTP). After tracepoint start, , KGTP will do these actions (Some of them will collect data and save them to tracepoint frame buffer) when Linux kernel execution to there addresses. After that, Linux kernel will keep execution.

KGTP supply some interfaces that GDB or other programe can take the data of tracepoint frame buffer out to parse.

About these interfaces, this doc have introduced "/sys/kernel/debug/gtp". And will introduce "/sys/kernel/debug/gtpframe" and "/sys/kernel/debug/gtpframe_pipe" later.

set tracepoint

The trace command is very similar to the break command. Its argument location can be a source line, a function name, or an address in the target program. The trace command defines a tracepoint, which is a address or some addresses that KGTP do some actions in it.

Howto handle the function is there but set tracepoint on it got fail

GCC will inline some static function to increase the performance. You cannot set tracepoint on the function name because object file doesn't have symbol of inline function.

How to set tracepoint condition

Like breakpoints, we can set conditions on tracepoints. The speed of tracepoints is faster than breakpoints because KGTP can do all the condition checks.

And you can use GDB command "condition" to specify the condition of a tracepoint. GDB command "condition N COND" will specify tracepoint number N to trace only if COND is true.

Value of $bpnum is the last ID of GDB tracepoint. Then you can use GDB command "condtion" set the condition of last tracepoint without get its ID. For example:

How to handle error "Unsupported operator (null) (52) in expression."

If you use condition about string, you will got this error when you call "tstart".

To handle it, you can convent the char to int to handle this issue, for example:

actions [num]

This command will prompt for a list of actions to be taken when the tracepoint is hit. If the tracepoint number num is not specified, this command sets the actions for the one that was most recently defined (so that you can define a tracepoint and then say actions without bothering about its number). You specify the actions themselves on the following lines, one action at a time, and terminate the actions list with a line containing just end. So far, the only defined actions are collect, teval, and while-stepping.

collect expr1, expr2, ...

Collect values of the given expressions when the tracepoint is hit. This command accepts a comma-separated list of any valid expressions. In addition to global, static, or local variables, the following special arguments are supported:

Please note that collect an pointer (collect ptr) will just collect the address of this pointer. Add a * before ptr will make action collect the data that pointer point to(collect *ptr).

teval expr1, expr2, ...

Evaluate the given expressions when the tracepoint is hit. This command accepts a comma-separated list of expressions. The results are discarded, so this is mainly useful for assigning values to trace state variables (see Simple trace state variables) without adding those values to the trace buffer, as would be the case if the collect action were used.

while-stepping n

Start and stop the tracepoint

Enable and disable the tracepoint

Like breakpoint, tracepoint can be control by GDB commands "enable" and "disable". But please note that it only useful when tracepoint stop.

Use tfind select the entry inside the trace frame info

GDB command "tfind" is used to select a entry of trace frame bufffer when tracepoint stop.

When GDB inside "tfind" mode, it will just show the values of this entry that the tracepoint action collect. So it will output some error when print some values that action doesn't collect for example the argument of function. That is not a bug, please don't worry about it.

How to handle error "No such file or directory."

When GDB cannot find the source code of Linux kernel, it will show this error message. For example:

You can use GDB command "set substitute-path" to handle it. The prev example, the Linux kernel source is in "/build/buildd/test/linux-3.2.0/". But vmlinux let GDB find Linux kernel source in "/build/buildd/linux-3.2.0/". You can handle it with:

Save the trace frame info to a file

/sys/kernel/debug/gtpframe supplies trace frame in tfile format (GDB can parse it) when KGTP is stop.

Please note that some "cp" cannot handle it very well, please use "cat /sys/kernel/debug/gtpframe > ./gtpframe" to copy it.

Show and save the tracepoint

You can use GDB command "save tracepoints filename" to save the commands that setup the tracepoints and actions into file filename. Then you use use GDB commands "source filename" to setup this tracepints again.

Delete tracepoint

GDB command "delete id" will delete tracepoint id. If "delete" without argument, it will delete all the tracepoint.

Use tracepoint get register info from a point of kernel

The following is an example that records the value of all registers when "vfs_readdir" is called.

Use tracepoint get the value of variable from a point of kernel

The following is an example that records the value of "jiffies_64" when the function "vfs_readdir" is called:

Show all the traced data of current frame

Get status of tracepoint

Set the trace buffer into a circular buffer

The frame buffer is not a circular buffer by default. When the buffer is full, the tracepoint will stop.

Following command will set frame buffer to a circular buffer. When the buffer is full, it will auto discard traceframes (oldest first) and keep trace.

Do not stop tracepoint when the GDB disconnects

KGTP will stop tracepoint and delete the trace frame when GDB disconnects with it by default.

Following command will open the KGTP disconnect-trace. After that, when GDB disconnects with KGTP, KGTP will not stop tracepoint. And after GDB reconnects to KGTP, it can keep control of KGTP like nothing happened.

kprobes-optimization and the execution speed of tracepoint

The tracepoint is execution together with Linux kernel. So it speed will affect the speed the system.

The KGTP tracepoint base on Linux kernel kprobe. Because the normal kprobe base on breakpoint instruction, so it is not very fast.

But if arch of kernel is X86_64 or X86_32 and kernel config didn't open "Preemptible Kernel" (PREEMPT), the kprobe is speed up by kprobes-optimization (CONFIG_OPTPROBES) that make kprobe very fast.

Please note that some KGTP functions will make this tracepoint use simple kprobe even if this Kernel support kprobes-optimization. This doc will add note when introduce these functions. Please avoid using them when you really care about the tracepoint speed.

How to use trace state variables

TSV can be accessed in tracepoint action and condition or direct access by GDB command.

Please note that just GDB 7.2.1 and later versions support use trace state variables directly, the old version of GDB can show the value of trace state variables through command "info tvariables".

Simple trace state variables

Trace state variable $c is created with initial value 0. The following action uses $c to count how many irqs happened in the kernel.

Also, you can set a value of variable to trace state variable, but don't forget covert variable to "uint64_t".

When using tfind, you can parse the trace frame buffer. If the value of a trace state variable is collected, you can parse it out.

If need, the tracepoint action that access the simple trace state variables will auto lock a spin lock for trace state variables. So it can handle race condition issue about trace state variables.

The following example is OK even if it running a machine that have more than one CPU.

Per_cpu trace state variables

When tracepoint action access to it, it will access to this CPU special trace state variables.

1. The tracepoint actions that access to per_cpu trace state variables don't have the race conditon issue. So it don't need lock the spin lock for trace state variables. It is faster than simple trace state variables on multi-core machine.

2. Write the action that count some CPU special thing with it is easier than simple trace state variables.

How to define

Local CPU variables

When access this trace state variable in tracepoint actions, it will return the variable's value of CPU that this tracepoint actions running on.

CPU id variables

When access this trace state variable in tracepoint actions or GDB command line, it will return the variable's value of CPU CPU_id.

Follow example can auto define a CPU id variables for each CPU of this machine. (Please note that need let GDB connect to KGTP before use these commands.)

Example 1

This example define a tracepoint that count the times that call vfs_read of each CPU.

Example 2

This example record stack dump of the function that close IRQ longest time of each CPU.

KGTP special trace state variables $current_task, $current_thread_info, $cpu_id and $clock can very easy to access to some special value. You can see them when GDB connects to the KGTP. You can use them in tracepoint conditions or actions.

Access $current_task in tracepoint condition and action will get that returns of get_current().

Access $current_task_pid in tracepoint condition and action will get that returns of get_current()->pid.

Access $current_thread_info in tracepoint condition and action will get that returns of current_thread_info().

Access $cpu_id in tracepoint condition and action will get that returns of smp_processor_id().

Access $clock in tracepoint condition and action will get that returns of local_clock() that return the timestamp in nanoseconds.

$rdtsc is only available on X86 and X86_64 architecture. Access it in anytime will get current value of TSC with instruction RDTSC.

Access $hardirq_count in tracepoint condition and action will get that returns of hardirq_count().

Access $softirq_count in tracepoint condition and action will get that returns of softirq_count().

Access $irq_count in tracepoint condition and action will get that returns of irq_count().

And KGTP has other special trace state variables $dump_stack, $printk_level, $printk_format and $printk_tmp. All of them output their values directly, as can be seen in Howto let tracepoint output value directly.

The following example counts in $c how many vfs_read calls that process 16663 does and collects the struct thread_info of current task:

sys_read() execute 3255 times in cpu0 and 1904 times in cpu1. Please note that this example just to howto use $cpu_id. Actially, this example use per_cpu trace state variables is better.

Special trace state variable $self_trace

$self_trace is different with the special trace state variables in the previous section. It is used to control the behavior of tracepoint.

In default, when tracepoint is triggered, the actions will not execute if the current_task is the a KGTP self process (GDB, netcat, getframe or some others process that access to the interface of KGTP).

If you want tracepoint actions execute with any task, please include a command access to the $self_trace in the actions i.e. add following command to the actions:

Trace the function return with $kret

Sometime, set the tracepoint to the end of function is hard because the Kernel is compiled with optimization. At this time, you can get help from $kret.

$kret is a special trace state variable like $self_trace. When you set value of it inside the action of tracepoint, this tracepoint be set with kretprobe instead of kprobe. Then it can trace the end of this function.

Please note that this tracepoint must set in the first address of the function in format "function_name".

Use $ignore_error and $last_errno to ignore the error of tstart

But sometime we need ignore this error and let KGTP keep work. For example: If you set tracepoint on the inline function spin_lock. This tracepoint will be set to a lot of addresses that some of them cannot be set kprobe. It will make tstart get fail. You can use "$ignore_error" ignore this error.

Use $cooked_clock and $cooked_rdtsc the time without KGTP used

Access these two trace state variables can get the time without KGTP used. Then we can get more close to really time that a part of code used even if the actions of tracepoint is very complex. They will be introduce in Cookbook (coming soon).

Use $xtime_sec and $xtime_nsec get the timespec

Access these two trace state variables will return the time of day in a timespec that use getnstimeofday.

Howto backtrace (stack dump)

Each time your program performs a function call, information about the call is generated. That information includes the location of the call in your program, the arguments of the call, and the local variables of the function being called. The information is saved in a block of data called a stack frame. The stack frames are allocated in a region of memory called the call stack.

Collect stack with $bt and use GDB command "backtrace"

Because this way is faster (just collect the stack when trace) and parse out most of info inside the call stack (it can show all the stack info that I introduce). So I suggest you use this way to do the stack dump.

~~The general collect the stack command in GDB tracepoint is: In x86_32, following command will collect 512 bytes of stack.~~

~~These commands is so hard to remember, and the different arch need different command.~~

KGTP have an special tracepoint trace state variable $bt. If tracepoint action access it, KGTP will auto collect the $bt size (default value is 512) stack. For example, this command will collect 512 bytes stack memory:

If you want to change size of $bt, you can use following GDB command before "tstart":

Following part is an example about howto collect stack and howto use GDB parse it:

You can see that when you use up, down or frame to the different calll stack frame, you can output the value of the arguments and local variables of different call stack frame.

Collect stack of current function's caller with $_ret

If you just want to collect stack of current function's caller, please use $_ret.

Please note that set the tracepoint that collect $_ret cannot in the first address of function.

You see that the caller of function vfs_read is sys_read. And the local variable ret of sys_read is -9.

Use $dump_stack to output stack dump through printk

Because this way need parse the stack when tracing and call printk inside, so it will be slow, unsafe, unclear and cannot access a lot of info of call stack. So I suggest you use the prev way to do stack dump.

KGTP has special trace state variable $dump_stack, "collect" it will let Linux Kernel output stack dump through printk.

Howto let tracepoint output value directly

In the previous parts, you may understand that to get a value from Linux kernel, you need to use a tracepoint "collect" action to save the value to the tracepoint frame and use the GDB command "tfind" to parse the value from the frame data.

But we want get the value directly sometimes, so KGTP supports two ways to output values directly.

Switch collect to output the value directly

KGTP has special trace state variables $printk_level, $printk_format and $printk_tmp to support this function.

$printk_level: if its value is 8 (this is the default value), "collect" action will save value to the tracepoint frame in the simple behavior.

If its value is 0-7, "collect" will output the value through "printk" directly, and value will be the level of printk. The level is:

$printk_format, collect printk will output value in the format that is set by it. The format is:

Following example shows a count number, pid, jiffies_64 and the file name that call vfs_readdir:

Howto use watch tracepoint control hardware breakpoints to record memory access

Watch tracepoint can control hardware breakpoints to record the memory access through set some special trace state variables in its action.

Please note that watch tracepoint is just support by X86 and X86_64 now. And dynamic watch tracepoint just can work OK in Linux 2.6.27 and newer version because Linux 2.6.26 and older version have some IPI issues on smp support.

Trace state variables of watch tracepoint

Static watch tracepoint

You can use static watch tracepoint when you want watch value of a global variable or some memory that you can get its address directly. Following example is watch jiffies_64's write:

Dynamic watch tracepoint

If you want to watch value of a local variable or some memory that you just get get its address inside the function, you can use dynamic watch tracepoint. Following example is watch write of f->f_pos and f->f_op inside function get_empty_filp:

Define a dynamic watch tracepoint. The address "1" of it is not the address of memory that it will watch. It just help tracepoint that setup this dynamic watch tracepoint can find it.

Define a normal tracepoint that start to watch f->f_pos and f->f_op inside function get_empty_filp.

Define a normal tracepoint that stop the tracepoint that watch file->f_pos and file->f_op.

Use while-stepping let Linux kernel do single step

Howto use while-stepping

while-stepping is a special tracepoint action that include some actions with it.

When tracepoints that its actions include "while-stepping n" execute, it will do n times single steps and executes the actions of while-stepping. For example:

Please note that tracepoint will disable the interrupt of current CPU when it do single step. Access $step_count in actions will get the count of this step that begin with 1.

Read the traceframe of while-stepping

Or you can switch KGTP to replay mode to select all the traceframe of a while-stepping tracepoint with GDB execution and reverse-execution commands. For example:

Set breakpoints (Just valid in replay mode, will not affect Linux kernel execution).

Howto show a variable whose value has been optimized away

That is because value of inode and res is optimized. Linux Kernel is built with -O2 so you will get this trouble sometimes.

Update your GCC

Get the way that access the variable that has been out through parse ASM code

Even if update the GCC to the newer version, you will still meet the issue. The main reason is the data is inside the registers but GCC doesn't put it to debug info. Then GDB just can output this variable has been optimized away.

But you can get where is the variable from ASM code and access it inside the tracepoint actions.

Following is a example that find variable "f" of function get_empty_filp and use it in tracepoint actions:

Now use "disassemble /m" command get the ASM code and source line that have relation with "f" and parse them.

Code from "+98" to "+132" is not show in this part because they belong to other inline function. But you can get them with GDB command "disassemble get_empty_filp".

According to the ASM code you can see that return value of kmem_cache_alloc is inside $rax and its value is set to $rbx.

Set a value to element of f, the ASM code is set value of $r12 to a address that base address is $rbx. It also looks like $rbx is "f".

How to get the function pointer point to

If the debug info of the function pointer is not optimized out

If the debug info of the function pointer is optimized out

Please note that while-stepping will make tracepoint cannot use kprobes-optimization.

/sys/kernel/debug/gtpframe and offline debug

/sys/kernel/debug/gtpframe supplies trace frame in tfile format (GDB can parse it) when KGTP is stop.

Then you can find files gtpstart and gtpstop in current directory. Copy it to the machine that you want to debug.

In the debugged machine, copy the program "putgtprsp" and "gtp.ko" in the KGTP directory to this machine first. After insmod the gtp.ko:

If you want to save the value to the trace frame and parse later, you can use file "/sys/kernel/debug/gtpframe" that has the trace frame. Copy it to the PC that has GDB.

Please note that some "cp" cannot handle it very well, please use "cat /sys/kernel/debug/gtpframe > ./gtpframe" to copy it.

Please note that if you want connect KGTP from GDB in remote machine after use offline debug, you need "rmmod gtp" and "insmod gtp.ko" before call "nc".

How to use /sys/kernel/debug/gtpframe_pipe

This interface supplies same format trace frame with "gtpframe". But it can work when KGTP is running. After data is read, it will auto deleted from trace frame like "trace_pipe" of ftrace.

Get the frame info with GDB

This way is better to work with python to parse Kernel. add-ons/hotcode.py is an example of python script.

Get the frame info with cat

Get the frame info with getframe

KGTP package include a program "getframe" can help you save the trace frame to files.

Use $pipe_trace

For the lock safe, KGTP will ignore the task that read the /sys/kernel/debug/gtpframe_pipe in default.

If you really need trace this task, and be sure that is safe. You can use following command before call "tstart":

Use KGTP with user applications

Let GDB connect KGTP for user applications

2) If user applications is running on the current machine, use GDB command "target extended-remote /sys/kernel/debug/gtp" connect to KGTP. If user applications is running on the remote machine, use netcat like GDB on remote machine but replace "target remote" to "target extended-remote".

3) Load user applications (it must be built with GCC option "-g" to make it has debug information) with GDB command "file".

Read memory of user applications directly

After GDB attach the user applications success, you can access the memory of this task with GDB commands "p" and "x". You can get help of these commands with GDB commands "help p" and "help x". For example:

Trace user applications

KGTP use uprobes function of Linux kernel trace user applications, just Linux kernel 3.9 and later version support this function.

The build config of most Linux distributions's Linux kernel (3.9 and later) has opened uprobes.

If current Linux kernel uprobes is opened, you can set tracepoint according to GDB tracepoint after GDB attach the user applications success. For example:

Please note that even if you just attch one of these tasks, user applications's tracepoint will be triggered by all tasks of a user application. (I think this is a very interesting feature of uprobe, so I didn't limit it in KGTP tracepoint.)

You can add $current_task_pid check to conditions of tracepoint to make tracepoint just be triggered by one of this task. Following example is set a tracepoint that just for task 985:

And you can "collect $current_task_pid" in tracepoint actions to make sure which task triggers the tracepint. For example:

collect stack (for backtrace) of system from Linux kernel to user applications in tracepoint

$current is a special trace state variable that if the action of an tracepoint access it, this tracepint will access the values of the registers and the memory of current task instead of Linux kernel.

In general, the tracepoint will get the registers value of current task from task_pt_regs. Then collect $current in tracepoint actions will let this tracepoint access values of current task. For example:

In addition, for some special function that its arguments include the pointer to the registers(for example: do_IRQ function of X86), tracepoint need get the registers from the arguments of fuction. Then set the pointer to $current will let this tracepoint get it. For example:

$current_task_user is a special trace state variable that it is value will be true when current task is in user mode.

With these two trace state variables, you can use KGTP collect the stack(backtrace) of current task.

Following example show how we do backtrace(stack dump) from user space to Linux kernel:

How to use add-ons/hotcode.py

This script can show the hottest code line in the Linux kernel or user space program through parse and record the pc address in the interrupt handler.

How to add plugin in C

API

These two functions register and unregister the plugin module. Then when KGTP will add module usage count when it access the resource of the plugin module.

When rmmod the plugin module, use this function remove the TSV that gtp_plugin_var_add add.

Example

plugin_example.c that in the KGTP directory is the example for KGTP plugin. You can use "make P=1" build it. It add 4 TSV to KGTP.

How to use

How to use performance counters

Performance counters are special hardware registers available on most modern CPUs. These registers count the number of certain types of hw events: such as instructions executed, cachemisses suffered, or branches mis-predicted - without slowing down the kernel or applications. These registers can also trigger interrupts when a threshold number of events have passed - and can thus be used to profile the code that runs on that CPU.

The Linux Performance Counter subsystem called perf event can get the value of performance counter. You can access it through KGTP perf event trace state variables.

Please goto read the file tools/perf/design.txt in Linux Kernel to get more info about perf event.

Define a perf event trace state variable

Define a per_cpu perf event trace state variable

Please note that if you define a per_cpu perf event trace state variable, you will not need define the cpu id("pe_cpu") because KGTP already get it.

The perf event type and config

If the type is 3(PERF_TYPE_HW_CACHE), the config need to divide to 3 parts: First one is cache id, it need be << 0 before set to config:

If you want get the perf count of PERF_COUNT_HW_CACHE_L1I(1), PERF_COUNT_HW_CACHE_OP_WRITE(1) and PERF_COUNT_HW_CACHE_RESULT_MISS(1), you can use:

tools/perf/design.txt in Linux Kernel have more info about type and config of perf event.

Enable and disable all the perf event in a CPU with $p_pe_en

I think the best way that count a part of code with performance counters is enable all the count in the begin of the code and disable all of them in the end. You can do it with "pe_en". But if you have a lot of perf event trace state variables. That will make the tracepoint action very big. $p_pe_en is for this issue. You can enable all the perf event trace state variables in current CPU with following action:

GDB scripts to help with set and get the perf event trace state variables

Following is a GDB script define two commands dpe and spe to help define and show the perf event trace state variables.

You can put it to the ~/.gdbinit or your tracepoint script. Then you can use this two commands in GDB directly.

Following is an example to use it get the performance counters of function tcp_v4_rcv:

Appendix A Preparatory work before use KGTP

Linux kernel

If your system use the Linux kernel that is built by yourself

If use with Linux kernel of Android

The default Linux kernel config of Android should not support KGTP. To use KGTP, Linux kernel of Android need open following options:

Please rebuild your Linux kernel if you change any options of the Linux kernel config.

If your system use the Linux kernel from distribution

Ubuntu

The standard method of install the Linux kernel debug image

Create an /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:

Stable releases (not alphas and betas) require three more lines adding to the same file, which is done by the following terminal command:

Then you can find Linux kernel debug image in "/usr/lib/debug/boot/vmlinux-$(uname -r)".

Please note that this step Get Linux kernel debug image need do again when Linux kernel update.

The second method of install the Linux kernel debug image

If you got some trouble with the standard method, please use following commands to install the Linux kernel debug image.

Install the Linux kernel headers

Install the Linux kernel source

New way

Old way

Fedora

Install the Linux kernel debug image

Then you can find Linux kernel debug image in "/usr/lib/debug/lib/modules/$(uname -r)/vmlinux".

Install the Linux kernel devel package

Please note that after update the Linux kernel package, you may need to call this command.

Make sure current Linux kernel debug image is right

GDB open the right Linux kernel debug image is an very important because GDB will get the debug info and address info from it. So before you use KGTP, please do the check to make sure about it.

There are 2 ways to do the check, what I suggest is do both of them to make sure Linux kernel debug image is right.

Where is the current Linux kernel debug image

If you build Linux kernel with yourself, file "vmlinux" in the Linux kernel build directory is the debug image.

Use /proc/kallsyms

In the system that its Linux kernel is what you want to trace, use following command to get the address of sys_read and sys_write:

Then we can get that the address of sys_read is 0xffffffff8117a520 and the address of sys_write is 0xffffffff8117a5b0.

After that use GDB get address of sys_read and sys_write from Linux kernel debug image:

The address of sys_read and sys_write is same, so the Linux kernel debug image is right.

Use linux_banner

After that, connect to KGTP following the way in Make GDB connect to gtp connect to KGTP and print linux_banner again.

This linux_banner is the kernel info that Linux kernel that KGTP is tracing. If it is same with the prev kernel info, the Linux kernel debug image is right.

Handle the issue that Linux kernel debug image's address info is not same with Linux kernel when it running

In X86_32, you will found that the Linux kernel debug image's address info is not same with Linux kernel when it running through the ways in Make sure current Linux kernel debug image is right. And you determine the Linux kernel debug image is right.

The values of these two options are different. Please note that the "Physical address where the kernel is loaded" is not showed in config sometimes. You can get its value through search "PHYSICAL_START".

You can handle this issue through change "Alignment value to which kernel should be aligned" same with "Physical address where the kernel is loaded".

Get KGTP

Get KGTP through http

Get KGTP through git

Mirrors

Config KGTP

Following part is the default config of KGTP inside the Makefile. With this config, KGTP will build together with current kernel that running on this machine.

KERELDIR is set to the directory which holds the kernel you want to build for. By default, it is set to the kernel that you are running.

Please note that this directory should be Linux kernel build directory or linux-headers directory but not the Linux kernel source directory. And the Linux kernel build directory should be used after build successful.

CROSS_COMPILE is set to the prefix name of compiler that you want to build KGTP. Empty to compile with your default compiler.

Or you can choose which kernel you want build with and which compiler you want use by change Makefile.

KERNELDIR is set to /home/teawater/kernel/bamd64. Compiler will use x86_64-glibc_std-gcc.

Compile KGTP

Normal compile

Handle the compile errors

Compile KGTP with some special config

Most of time, KGTP can auto select right options to build with Various versions of Linux kernel.

But if you want config special options with yourself, you can read following part:

Install and uninstall KGTP

KGTP don't need to be install because it can insmod directly inside its directory (See Insmod the KGTP module). But if you need, you can install it to your system.

Use KGTP with DKMS

Use KGTP patch for Linux kernel

Most of time, you don't need KGTP patch because KGTP can build as a LKM and very easy to use. But to help some people include KGTP to them special Linux Kernel tree, KGTP supply patches for Linux kernel.

Install GDB for KGTP

The GDB that older than 7.6 have some bugs of tracepoint. And some functions of GDB are not very well.

So if your GDB is older than 7.6 please go to https://code.google.com/p/gdbt/ to get howto install GDB for KGTP. It supplies sources of UBUBTU, CentOS, Fedora, Mandriva, RHEL, SLE, openSUSE. Also have static binary for others.

Appendix B How to let GDB connect to KGTP

Normal Linux

Insmod the KGTP module

Handle the issue that cannot find "/sys/kernel/debug/gtp"

Maybe you will got some error for examle "sysfs is already mounted on /sys". Please ignore it.

Make GDB connect to gtp

Load Linux kernel debug image to GDB

GDB on the current machine

GDB on remote machine

Android

Insmod the KGTP module

Directory "/" may be read-only. You can choice other directory or use command "sudo adb shell mount -o rw,remount /" remount the directory to can write.

Handle the issue that cannot find "/sys/kernel/debug/gtp"

If you got this issue, please make sure "Debug Filesystem" is opened in your kernel config first. Please goto If_your_system_use_the_Linux_kernel_that_is_built_by_yourself see howto “If use with Linux kernel of Android” see howto open it.

Maybe you will got some error for examle "Device or resource busy". Please ignore it.

GDB connect to the KGTP

Appendix C Add module symbols to GDB

Add symbols with hand is not very easy, so KGTP package include an GDB python script "getmod.py" and a program "getmod" can help you.

How to use getmod

"getmod" is written by C so you can use it anywhere even if in an embedded environment.

If you use remote debug or offline debug, maybe you need change the base directory. Following example is for it.

Function	GDB debug normal program	GDB control KGTP debug Linux kernel
Preparatory work	Have a GDB installed in your system. Program built with "-g".	Quick config and start KGTP
Attach	Use command "gdb -p pid" or GDB command "attach pid" can attach a program that running in the system.	Quick config and start KGTP
Breakpoints	GDB command "b place_will_stop", let program execute after this command. Then programe will stop in the place that setup a breakpoint.	KGTP doesn't support breakpoints but it support tracepoints. Tracepoints can be considered as a special kind of breakpoints. It can be setup in some place of Linux kernel and define some commands that you want to do in its actions. When tracepoints start, they will execute these commands when Linux kernel execute to these place. When tracepoint stop, you can use some GDB commands parse the data that get by tracepoints like what you do when program stop by breakpoints. Difference is breakpoints will stop the program But the tracepoints of KGTP not. Please goto GDB tracepoint get howto use it.
Memory read	After GDB stop the program(maybe doesn't need), it can read memory of program with GDB command "print", "x" and so on.	You can set special actions to collect memory to traceframe in tracepoints, and get the its value when tracepoint stop.collect expr1, expr2, ... Use tfind select the entry inside the trace frame info Or you can read memory directly when Linux kernel or program is running.Direct access the current value in normal mode
Step and continue	GDB can continue program execution with command "continue" and stop it with CTRL-C.	KGTP never stop the Linux kernel. But tracepoint can be start and stop.Start and stop the tracepoint Or use while-stepping tracepoint record Linux kernel with some times single step and Let KGTP switch to replay mode. Then it support execution commands (continue, step) and reverse-execute commands (reverse-continue, reverse-step). Use while-stepping let Linux kernel do single step
Backtrace	GDB can print backtrace of all stack frames with command "backtrace".	KGTP can do it too.Howto backtrace (stack dump)
Watchpoint	GDB can let programe stop when some memory access happen with watchpoint.	KGTP can record the memory access with watch tracepoint. Howto use watch tracepoint control hardware breakpoints to record memory access
Call function	GDB can call function of program with command "call function(xx,xx)".	KGTP can call function of Linux kernel with plugin.How to add plugin in C

Name	Written by normal tracepoint	Read by normal tracepoint	Written by static watch tracepoint	Read by static watch tracepoint	Written by dynamic watch tracepoint	Read by dynamic watch tracepoint
$watch_static	Not support	Not support	If "teval $watch_static=1", then this tracepoint is static watch tracepoint.	Not support	If "teval $watch_static=1", then this tracepoint is static watch tracepoint.	Not support
$watch_set_id	When this tracepoint want to setup a dynamic watch tracepoint, set a id of a dynamic watch tracepoint to $watch_set_id to point out which dynamic watch tracepoint you wan to setup.	Not support	Not support	Not support	Not support	Not support
$watch_set_addr	When this tracepoint want to setup a dynamic watch tracepoint, set the address of a dynamic watch tracepoint to $watch_set_addr to point out which dynamic watch tracepoint you wan to setup.	Not support	Not support	Not support	Not support	Not support
$watch_type	When this tracepoint want to setup a dynamic watch tracepoint, set the watch type of this dynamic watch tracepoint to $watch_type. 0 is exec. 1 is write. 2 is read or write.	Get the value that this tracepoint set to $watch_type	Set the type of this watch tracepoint.	Get the type of this watch tracepoint.	Set the default type of this watch tracepoint.	Get the type of this watch tracepoint when it really exec.
$watch_size	When this tracepoint want to setup a dynamic watch tracepoint, set the watch size of this dynamic watch tracepoint to $watch_size. The size should be 1, 2, 4, 8.	Get the value that this tracepoint set to $watch_size.	Set the size of this watch tracepoint.	Get the size of this watch tracepoint.	Set the default size of this watch tracepoint.	Get the size of this watch tracepoint when it really exec.
$watch_start	Set the address to a dynamic watch tracepoint(set by $watch_set_addr or $watch_set_id) and let it try to start work.	Get the result of this start. (It will fail becasue X86 just have 4 hardware breakpoints.) Get 0 if success. If < 0 is the error id.	Not support	Not support	Not support	Not support
$watch_stop	Set a address to $watch_stop will let a dynamic watch tracepoint that watch in this address stop.	Get the result of this stop.	Not support	Not support	Not support	Not support
$watch_trace_num	Not support	Not support	Not support	Not support	Not support	The tracepoint number that setup this dynamic watch tracepoint.
$watch_trace_addr	Not support	Not support	Not support	Not support	Not support	The tracepoint address that setup this dynamic watch tracepoint.
$watch_addr	Not support	Not support	Not support	The address that this watch tracepoint is watching.	Not support	The address that this watch tracepoint is watching.
$watch_val	Not support	Not support	Not support	The current value of the memory that this watch tracepoint is watching.	Not support	The current value of the memory that this watch tracepoint is watching.
$watch_prev_val	Not support	Not support	Not support	The previous value of the memory that this watch tracepoint is watching.	Not support	The previous value of the memory that this watch tracepoint is watching.
$watch_count	Not support	Not support	Not support	Not support	Not support	A special count for this watch tracepoint session.