imx8mp: Linux 5.4.70: USB Disconnect causes exception (OOPS) in usb_hcd_check_unlink_urb()

[mahesh_hns]

Hello,

I have a imx8mp running linux 5.4.70 connected to a USB modem. When running a test case where the USB2 is powered off (manully through linux commandline or powering the USB modem off forcibly) many times, once in a while the linux kernel throws exception as below

1577.892050] Mem abort info:
[ 1577.894844] ESR = 0x96000006
[ 1577.897897] EC = 0x25: DABT (current EL), IL = 32 bits
[ 1577.903208] SET = 0, FnV = 0
[ 1577.906259] EA = 0, S1PTW = 0
[ 1577.909400] Data abort info:
[ 1577.912279] ISV = 0, ISS = 0x00000006
[ 1577.916113] CM = 0, WnR = 0
[ 1577.919081] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000058663000
[ 1577.925521] [0000000000000001] pgd=00000000586a9003, pud=00000000586ac003, pmd=0000000000000000
[ 1577.934223] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 1577.939795] Modules linked in: mxl_moca_ctrl(O)
[ 1577.944332] CPU: 0 PID: 1503 Comm: kworker/0:0 Tainted: G W O 5.4.70-2.3.0-00011-g66b1d56-dirty #40
[ 1577.954504] Hardware name: SSMv2 Board (DT)
[ 1577.958694] Workqueue: usb_hub_wq hub_event
[ 1577.962880] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 1577.967673] pc : usb_hcd_check_unlink_urb+0x194/0x430
[ 1577.972727] lr : usb_hcd_check_unlink_urb+0x178/0x430
[ 1577.977776] sp : ffff8000123fb7c0
[ 1577.981089] x29: ffff8000123fb7d0 x28: ffff00001d2aa000
[ 1577.986404] x27: ffff00001d2a0810 x26: ffff800010ff31d8
[ 1577.991716] x25: ffff00001dae7a18 x24: 00000000ffffff94
[ 1577.997028] x23: ffff800011b09000 x22: ffff80001133b9f0
[ 1578.002340] x21: ffff800011b091a0 x20: ffff00001dae7a00
[ 1578.007651] x19: 0000000000000001 x18: 0000000000000030
[ 1578.012963] x17: 0000000000000000 x16: 0000000000000000
[ 1578.018275] x15: ffff80001187c000 x14: 3a7473694c206573
[ 1578.023587] x13: 7265766172542020 x12: 353332313a627275
[ 1578.028899] x11: 5f6b6e696c6e755f x10: 6b636568635f6463
[ 1578.034045] usb 2-1: usb_submit_urb:380 : dev->state < USB_STATE_UNAUTHENTICATED, returning -ENODEV
[ 1578.034212] x9 : 0000000000000000 x8 : 0000000000000007
[ 1578.043372] write urb submit err -19
[ 1578.048651] x7 : 00000000000285b5 x6 : 0000000000000001
[ 1578.048655] x5 : 0000000000000001 x4 : 0000000000000000
[ 1578.048659] x3 : ffff800010ff31d8 x2 : 109c0ec419c3fc00
[ 1578.048663] x1 : 109c0ec419c3fc00 x0 : ffff8000119e1a40
[ 1578.048667] Call trace:
[ 1578.048673] usb_hcd_check_unlink_urb+0x194/0x430
[ 1578.048681] xhci_urb_dequeue+0x58/0x500
[ 1578.057560] unlink1+0x44/0x188
[ 1578.057564] usb_hcd_flush_endpoint+0x1e8/0x498
[ 1578.057569] usb_disable_endpoint+0x64/0x248
[ 1578.057575] usb_disable_interface+0x64/0x100
[ 1578.068191] usb_unbind_interface+0x174/0x3d0
[ 1578.068197] device_release_driver_internal+0xf4/0x1c0
[ 1578.068201] device_release_driver+0x14/0x20
[ 1578.068207] bus_remove_device+0xcc/0x150
[ 1578.075960] device_del+0x13c/0x348
[ 1578.075964] usb_disable_device+0xac/0x2d0
[ 1578.075968] usb_disconnect+0xa4/0x3b8
[ 1578.075974] hub_port_connect+0x7c/0x740

After putting debug prints, it was found that the hcd->ep.urb_list sometimes has a NULL entry or an entry which, when accessed throws OOPS (perhaps freed memory, but non NULL). Has anyone faced this issue? or a similar one?

[mahesh_hns]

Thanks for the reply.

We use a custom board. The patch suggested doesn't apply to the older code baseline we are on (5.4.70_2.3.0). 

I am trying to get our changes to linux-imx-L5.15y and hit a snag with finding the latest tool chain for the same. Is there a download link for the imx-5.15 toolchain to build the linux kernel standalone?  The Linux user guide suggests building a SDK in yocto and building the tool chain that way, but I am running in to issues setting it up. Is there a download link for the tool chain to cross compile linux-5.15y?

[mahesh_hns]

Thanks for the reply.

Okay. As per the Linux User Guide, here is the procedure I followed on a Docker VM (ubuntu 20.04)

mkdir imx-yocto-bsp
cd imx-yocto-bsp
repo init -u https://github.com/nxp-imx/imx-manifest -b imx-linux-kirkstone -m imx-5.15.71-2.2.0.xml
repo sync
DISTRO=fsl-imx-xwayland MACHINE=imx8mp-lpddr4-evk source imx-setup-release.sh ./build-xwayland
bitbake core-image-minimal -c populate_sdk

docker_mradhakr@6138e6418416:/build_env/imx-yocto-bsp/build-xwayland$ bitbake core-image-minimal -c populate_sdk
Traceback (most recent call last):
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/bin/bitbake", line 37, in <module>
sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/main.py", line 378, in bitbake_main
return ui_module.main(server_connection.connection, server_connection.events,
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/ui/knotty.py", line 415, in main
params.updateToServer(server, os.environ.copy())
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer
raise Exception("Unable to update the server configuration with local parameters: %s" % error)
Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last):
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/command.py", line 90, in runCommand
result = command_method(self, commandline)
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/command.py", line 286, in updateConfig
command.cooker.updateConfigOpts(options, environment, cmdline)
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/cooker.py", line 535, in updateConfigOpts
self.reset()
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/cooker.py", line 1774, in reset
self.initConfigurationData()
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/cooker.py", line 404, in initConfigurationData
self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/bb/cooker.py", line 313, in add_filewatch
watcher.add_watch(f, self.watchmask, quiet=False)
File "/build_env/imx-yocto-bsp/sources/poky/bitbake/lib/pyinotify.py", line 1888, in add_watch
raise WatchManagerError(err, ret_)
pyinotify.WatchManagerError: add_watch: cannot watch /build_env/imx-yocto-bsp/build-xwayland/conf WD=-1, Errno=No space left on device (ENOSPC)

doc

There is about 324G free available , I still see the above error. Do you know how I get past this error?

[AldoG]

Hello,

searching on the web there are 2 things that you may try:

Increasing max user watches
$ sudo sysctl fs.inotify.max_user_watches=16384
https://github.com/sass/node-sass/issues/2534

clean up the Docker system
$ docker system prune
https://docs.docker.com/engine/reference/commandline/system_prune/

Best regards,
Aldo.

[mahesh_hns]

Thank you for the steps you mentioned. I was able to proceed with my Yocto build!

However, I am not sure itbuild the cross compilation toolchain

After  building with the above steps, the /tmp/depoy/sdk/ contains only the below

fsl-imx-xwayland-glibc-x86_64-core-image-minimal-armv8a-imx8mp-lpddr4-evk-toolchain-5.15-kirkstone.sh

When I ran it, 

You are about to install the SDK to "/disk2/mradhakr/fsl-imx-xwayland/5.15-kirkstone". Proceed [Y/n]? y
Extracting SDK..................................................................................................done
Setting it up...done
Your environment is misconfigured, you probably need to 'unset LD_LIBRARY_PATH'
but please check why this was set in the first place and that it's safe to unset.
The SDK will not operate correctly in most cases when LD_LIBRARY_PATH is set.
For more references see:
http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN80
http://xahlee.info/UnixResource_dir/_/ldpath.html
/disk2/mradhakr/fsl-imx-xwayland/5.15-kirkstone/post-relocate-setup.sh: Failed to source /disk2/mradhakr/fsl-imx-xwayland/5.15-kirkstone/environment-setup-armv8a-poky-linux with status 1
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /disk2/mradhakr/fsl-imx-xwayland/5.15-kirkstone/environment-setup-armv8a-poky-linux

Now, I was expecting aarch64-poky-linux which isn't there.  Did I use a wrong target/machine? (imx8mp-lpddr4-evk)

My intention is to build cross compilation tools (running on x86_64) to compile imx8mp linux standalone

[AldoG]

Hello,

Please try by using the following command:
$ unset LDFLAGS

Best regards,
Aldo.

[mahesh_hns]

Thanks,I'll try that. But my main concern is that I am not seeing the aarch64 file

/disk2/mradhakr/fsl-imx-xwayland/5.15-kirkstone/post-relocate-setup.sh: Failed to source /disk2/mradhakr/fsl-imx-xwayland/5.15-kirkstone/environment-setup-armv8a-poky-linux with status 1

The aarch64 file (mentioned in the user guide) does not get created. Am I building yocto with a wrong target machine?

[AldoG]

Hi,

No this should be correct, please try with the command:
$ unset LDFLAGS

And then install SDK again so it install all necessary files for standalone build.

Best regards,
Aldo.

[mahesh_hns]

Thanks, here is how it went.

mradhakr@gtowlnxvm17 /disk2/mradhakr/imx_5.15_SDK/toolchain $ unset LDFLAGS
mradhakr@gtowlnxvm17 /disk2/mradhakr/imx_5.15_SDK/toolchain $ unset LD_LIBRARY_PATH

mradhakr@gtowlnxvm17 /disk2/mradhakr/imx_5.15_SDK/toolchain $ ../fsl-imx-xwayland-glibc-x86_64-core-image-minimal-armv8a-imx8mp-lpddr4-evk-toolchain-5.15-kirkstone.sh
NXP i.MX Release Distro SDK installer version 5.15-kirkstone
============================================================
Enter target directory for SDK (default: /opt/fsl-imx-xwayland/5.15-kirkstone): /disk2/mradhakr/imx_5.15_SDK/toolchain
You are about to install the SDK to "/disk2/mradhakr/imx_5.15_SDK/toolchain". Proceed [Y/n]? Y
Extracting SDK..................................................................................................done
Setting it up...done
SDK has been successfully set up and is ready to be used.
Each time you wish to use the SDK in a new shell session, you need to source the environment setup script e.g.
$ . /disk2/mradhakr/imx_5.15_SDK/toolchain/environment-setup-armv8a-poky-linux

mradhakr@gtowlnxvm17 /disk2/mradhakr/imx_5.15_SDK/toolchain $ ls -la
total 40
drwxr-xr-x 3 mradhakr eng 4096 Feb 1 13:28 .
drwxr-xr-x 4 mradhakr eng 4096 Feb 1 13:26 ..
-rw-r--r-- 1 mradhakr eng 4143 Feb 1 13:27 environment-setup-armv8a-poky-linux
-rw-r--r-- 1 mradhakr eng 14027 Feb 1 13:27 site-config-armv8a-poky-linux
drwxr-xr-x 4 mradhakr eng 4096 Feb 1 13:27 sysroots
-rw-r--r-- 1 mradhakr eng 142 Feb 1 13:27 version-armv8a-poky-linux

Where is the environment-setup-aarch64-poky-linux , which I was looking for, to cross compile ? Is the ./fsl-imx-xwayland-glibc-x86_64-core-image-minimal-armv8a-imx8mp-lpddr4-evk-toolchain-5.15-kirkstone.sh produced by Yocto correct?

[AldoG]

Hello,

Yes it is correct, this will set the environmental variables needed for standalone compilation by using the following command:

$ source environment-setup-armv8a-poky-linux

Best regards,
Aldo.

[AldoG]

Hello,

Unfortunately no, we do not provide the toolchain it should be generate using Yocto, if you need help generating it do not hesitate to ask.

Best regards,
Aldo.

[AldoG]

Hello,

Would it be possible to try with latest Linux release?
Also, are you using your own custom board or is it an EVK?
If not, this seems like a similar issue as in this commit:
https://github.com/nxp-imx/linux-imx/commit/88d33c24076e4ec8a3251455baa69e62a7ebc539

Best regards,
Aldo.