Linux kernel/driver debugging problem

[grahamperkins]

Hello,

I am trying to use CodeWarrior to debug a linux kernel flash driver problem we have on a custom board (based on the TWR-LS1021A). I wish to single step through the code after hitting a breakpoint. I have two problem here:

1. I cannot see all the local variables.

2. Single stepping can jump back and forth.

These seem symptomatic of some level of optimisation being applied to the code. From web searches it appears that linux requires some level of optimisation to function correctly. Is this correct and if so how do people go about interactively debugging with such a restriction?

I am using CodeWarrior Studio ofr QorIQ LS series - ARM V7 ISA version 10.0.7 and NXP Yocto v1.8 SDK.

Thanks,

Graham.

[lunminliang]

Hello Graham,

I guess you need to follow steps as this one: Linux kernel and modules debug using CodeWarrior for ARMv7

Regards

Lunmin

[grahamperkins]

Hi Lunmin,

I followed that app note and debugged into a module. I still have the same problem with apparent optimisation causing erratic behaviour when single stepping.

Thanks,

Graham.

[marius_grigoras]

Hi Graham,

I’ve done some research (this is for ARMv8 SDK, but should be applicable as well for ARMv7) and I come up with a proposal for debugging kernel module in a more friendly way.

The goal is to compile the kernel module files without –O2 optimization flag…Or at least some of the file, as other files must have –O2 flag, as the compilation would fail without it.
Modifying the kernel Makefiles to remove –O2 flags wasn’t leading me to a solutions: by design, the kernel flags could not be changed in module. Module’s flags can only be appended to the existing kernel flags. Further investigation to change the kernel flags would result in a very intrusive solution, lot of changes, hard to reproduce on a different environment.

Therefore I was looking for a simpler solution: recompiling the module’s files from a script:

• No changes to the kernel or module sources or makefiles
• Not intrusive, very easy to reproduce on new environment

With this solution I was able to build mlx5_core.ko without optimization and debug the module accordingly.

Here are the steps to recompile the module files:
1. Create the environment to build the kernel and kernel modules:
a. Simpler solution: using yocto install with “bitbake virtual/kernel –c devshell”
Note: if build EAR5 from ISO, build folder is different than source folder. So in the linux “devshell” just go to the build folder:
cd /EAR5/Layerscape2-SDK-20150828-yocto/build_ls2085aqds_release/tmp/work/ls2085aqds-fsl-linux/linux-ls2-sdk/4.0-r0/build
b. Also is working manual build from kernel sources: in kernel sources export the compilation flags: CROSS_COMPILE and ARCH

2. Build the kernel module with the default flags:
a. make drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
Note: in order to build MLX5 I modified drivers/ethernet/mellanox/mlx5/core/Kconfig
config MLX5_CORE
tristate
depends on PCI
default m -> change default from n to m (build as module)
Mellanox driver should have been enabled from menuconfig (drivers->eth->drivers->Mellanox set to M (module))

b. or make modules to build all modules

3. Run the recompilation script (attached in the email).
The script has two mandatory parameters: kernel module files path and kernel module name (without .ko), see the example below. With these parameters it recompiles without –O2 all the kernel module’s files in that path.
The third parameters (optional) is interactive mode (-i): here the user is asked to choose for each file if is to be recompiled or not (for the case when a specific file could not be recompiled without –O2)

./recompile.sh drivers/net/ethernet/mellanox/mlx5/core mlx5_core

Note: the script is simple enough: it does not hardcode anything. I just read the “.<file>.o.cmd” files generated after step 2, get from the file the exact command used to build the file, just remove –O2 from that build command and run it again.

4. Repeat step 2.
This is needed to let the kernel makefiles to regenerate <module>.mod.c file and to build <module>.o, <module>.mod.o, <module>.ko

If you like the idea or have any comments or improvement ideas please let me know.

Please find an example below.
Example:

root@udp122517uds:ls2-linux# make drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CALL scripts/checksyscalls.sh
CC [M] drivers/net/ethernet/mellanox/mlx5/core/main.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/cmd.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/debugfs.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/fw.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/eq.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/uar.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/pagealloc.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/health.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/mcg.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/cq.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/srq.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/alloc.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/qp.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/port.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/mr.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/pd.o
CC [M] drivers/net/ethernet/mellanox/mlx5/core/mad.o
LD [M] drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.o
MODPOST 52 modules
CC drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.mod.o
LD [M] drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.koroot@udp122517uds:ls2-linux#
root@udp122517uds:ls2-linux#
root@udp122517uds:ls2-linux#
root@udp122517uds:ls2-linux# ./recompile.sh drivers/net/ethernet/mellanox/mlx5/core mlx5_core
Recompile alloc.o
Recompile cmd.o
Recompile cq.o
Recompile debugfs.o
Recompile eq.o
Recompile fw.o
Recompile health.o
Recompile mad.o
Recompile main.o
Recompile mcg.o
-----> Skip mlx5_core.ko
-----> Skip mlx5_core.mod.o
-----> Skip mlx5_core.o
Recompile mr.o
Recompile pagealloc.o
Recompile pd.o
Recompile port.o
Recompile qp.o
Recompile srq.o
Recompile uar.o
root@udp122517uds:ls2-linux#
root@udp122517uds:ls2-linux#
root@udp122517uds:ls2-linux# make drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CALL scripts/checksyscalls.sh
LD [M] drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.o
MODPOST 52 modules
CC drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.mod.o
LD [M] drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
root@udp122517uds:ls2-linux#

Regards,

Marius

[grahamperkins]

Hi Marius,

Thanks for the reply. I will try out your method and report back.

Thanks,

Graham.

[grahamperkins]

Marius,

Your method works fine. I have tailored it to my specific needs and can now single step through my driver.

Regards,

Graham

[addiyi]

Did you take a look on U-Boot Debug using CodeWarrior for QorIQ LS series – ARM V7 ISA ?

Adrian

[grahamperkins]

Adrian,

Yes I have as I used it to create the project. I set DEBUG_BUILD="1" in the machine conf file hoping that would disable optimisation and rebuilt. That appeared not to work.

Graham.