Moving boot to SPI flash. Having issues with DMA in mx233

[jlumme]

We have a project with imx233, based on Olinuxino's maxi board (iMX233-OLinuXino-MAXI).

Normally it's booting from microsd, but we would like to move to nand flash. We have ordered ( SST25VF064C) chips from Microchip, which are 64mbit SPI flash chips.

The plan is to hook this up to SPI2.

What other things we should think about, besides changing the jumpers to boot from microsd to "SPI2 flash master", and to flash a suitable linux kernel and rootfs to that nand? Do we need to do some configuration/recompiling to bootlets ? Are there some good recommendations on the flashing procedure ?

Thanks for any advice, or tips - this is a first time project for us, so any advice is welcome, thanks!

[tobiasdiendorfe]

Sorry for bringing a rather old thread back to life :smileylaugh:

We have build a board based on olinuxino maxi (imx233) and an "S25FL256SAGMFIR01" flash connected at spi2.

As bootloader we are using mainline "U-Boot 2015.10-rc2" and as linux distribution openwrt trunk from 09/2015 (kernel version 4.1.11). So the above patch (zero xfer_count int CTRL0) is already included (Double checked in code)!

Sadly we are facing stabillity issues we have tracked down to spi2 in combination with dma!

The kernel just freeze at some point. I have activated an "heartbeat" triggered led which just stops blinking, when the issue comes up. The kernel does'nt print any dump information after lock up.

Also it seems that the issue come up at random uptimes (~30sec to ~1h), maybe because we are using overlayfs and our app communicates with an also connected SC16IS752IPW (dual spi uart):

When the kernel locked up, on the ram lines (A4 & CLK) is some activity (massured with scope).

I have narowed the issue down the following way:

• remove complete spi2 ind DTS (including S25FL256 & also connected SC16IS752IPW)
  -> issue went away, uptime ~4 days!
• Switch spi-mxs.c driver to PIO mode only ( "if(1)" like mentioned above ), but without removing the spi2 (incl. devices)
  -> maybe issue went away? uptime currently ~2h on each of two devices

Are there any hints/patches regarding DMA which are not in the mainline kernel?

Is it safe to go with PIO only?

Somebody experienced similar issues of kernel lock up?

thx

best regards

Tobias

Our current SPI2 DTS:

			ssp1: ssp@80034000 {
				#address-cells = <1>;
				#size-cells = <0>;
				compatible = "fsl,imx23-spi";
				pinctrl-names = "default";
//				pinctrl-0 = <&spi2_pins_a>;
				cs-gpios = <&gpio0 3 0>, <&gpio0 4 0>;
				status = "okay";

				flash@0{
					compatible = "s25fl256s1";
					reg = <0>;
					spi-max-frequency = <4000000>;

m25p,fast-read;

					#address-cells = <1>;
					#size-cells = <1>;

					...
					partitions
					...
				};

				sc16is752: sc16is752@1 {
					compatible = "nxp,sc16is752";
					reg = <1>;
					spi-max-frequency = <4000000>;
					clocks = <&osc>;
					interrupt-parent = <&gpio0>;
					interrupts = <5 0x02>;
//					interrupts = <5 IRQ_TYPE_EDGE_FALLING>;
					gpio-controller;
					#gpio-cells = <2>;
				};
			};

[MarekVasut]

Just change the jumpers, write the u-boot.sb (or whatever you use, but it has to be the "bootstream" ... .sb ... file) to the SPI flash and light it up ... it will just work.

[jlumme]

Hi Marek,

Thanks for your reply! I hope you don't mind me writing a bit longer question for you.

Currently, we're not using uboot, but imx bootlets to boot from memory card. However,we're now planning to move to u-boot, since it's much more modern system, with an active development community.

You mention that we could move our bootstream to spi flash, but is it really that simple ?

I have couple of concerns:

- If we would move to u-boot, it's my understanding that uboot & spi on mx23 is not yet fully supported ? There is some work going on by Otavio Salvatore on this, but it doesnt seem to yet be complete.

- And if I would keep using the bootlets option from Freescale, I guess the currently elftosb2 tool generated sd_mmc_bootstream.raw can't be used as is in SPI flash ? Or maybe I'm still confused by something here, and I should study this area more.

Anyhow, I would prefer to move to u-boot, since it seems to have much more support, and could provide a good knowledge also for future.

In general, this may sound like a silly question, but I noticed you have done quite a bit of work in this arena (ie. usb loader fox mx23), so maybe you could give some hint: How do I specify "partitions" in SPI flash ? I know that in memory card it's a bit different because of that flash transition layer, and you can create normal partitions in fdisk for kernel and then rootfs, but for flash, and SPI flash in particularly, I'm not really yet sure...

So for example, first needs to be loader of course, let's say u-boot, then zImage with dtb, but how about rootfs ? How does kernel know where to find it ? Am I just confusing myself with irrelevant details ?

Thank you!

[fadilberisha]

- If we would move to u-boot, it's my understanding that uboot & spi on mx23 is not yet fully supported ?

You may consider using barebox bootloader. I submitted patches  for imx23-olinuxino boards,  and now imx23-olinuxino is included on barebox mainline. Source code contain short guide how to compile and prepare SD card. More detailed guide is  here:

imx23-olinuxino/barebox on olinuxino.md at master · koliqi/imx23-olinuxino · GitHub

Patches for SPI support on imx23-olinuxino board are here

Dropbox - 0001-ARM-imx23-olinuxino-Add-spi-support.patch - Simplify your life

Fadil Berisha

[jlumme]

Hi Fadil,

Thanks for your suggestion. I should try barebox definitely as well!  I guess in the end we don't really mind what bootloader we work with, if it works steadily, and can boot from SPI..

Though I do need to solve this SPI instability in linux side..

[MarekVasut]

BB is still depending on imx-bootlets ; U-Boot doesn't and that's the real problem here. The bootlets are broken and misconfigure the chip at times (there is a lot of ugly code in them, I won't go in beefy details to keep this short). Besides, the development of MX28 bootloader drivers happens mainly in U-Boot too, I'm not sure BB synces these with U-Boot on regular basis.

As for your kernel patch, check the upstream 3.7 , the SPI driver there is much better than the one in FSL BSP. For one it has fixed all the bugs with DMA (overrun when reading continuous data over the length of 1 DMA descriptor), it has fixed the issue where using /dev/mtdX directly crashed the kernel (due to vmalloc'd memory). Heck, it even has working the combined PIO/DMA mode where PIO is used for small transfers and DMA for large transfers to keep reasonable overhead and improve the performace ;-)

[fadilberisha]

>BB is still depending on imx-bootlets ;

Yes, barebox is depend on imx-bootlets and in my experience nothing wrong with that.

>U-Boot doesn't and that's the real problem here

No, real problem is that u-boot still does not support imx23-olinuxino board.

>I'm not sure BB synces these with U-Boot on regular basis.

As far as I know, barebox and u-boot are two independent boot loader projects, so no "synces" there.

I found that sometimes u-boot and sometimes barebox is ahead with support of new boards. I am using both of them, everything depend which one support board of my interest.  If needed to port new board, my choice is barebox because 1st it is  much easier to do and second,  I am not affiliated with neither of them.

>As for your kernel patch, check the upstream 3.7 , the SPI driver there is much better than the one in FSL BSP.

My patches have nothing to do with your  SPI driver nor driver  in FSL BSP. My patches only adding SPI support on imx23-olinuxino board and apply to imx23-olinuxino.dts file. They are already on kernel - check upstream 3.7.

>For one it has fixed all the bugs with DMA (overrun when reading continuous data over the length of 1 DMA descriptor), it has fixed the issue where using /dev/mtdX directly crashed the kernel (due to vmalloc'd memory). Heck, it even has working the combined PIO/DMA mode where PIO is used for small transfers and DMA for large transfers to keep reasonable overhead and improve the performace ;-)

You did great job. Thank you for hard work  and don't worry nobody is contesting your driver.

Fadil Berisha

[MarekVasut]

You mention that we could move our bootstream to spi flash, but is it really that simple ?

Yes -- just write it at 0x0 and boot, that's all (unless you need some complicated encryption setup or whatnot, then it might just be more annoying.

- If we would move to u-boot, it's my understanding that uboot & spi on mx23 is not yet fully supported ?

I'm almost certain the SPI IP block is pretty much the same on MX23 as is on MX28.

There is some work going on by Otavio Salvatore on this, but it doesnt seem to yet be complete.

True, Otavio is slacking on the MX23 support. But it's really easy to fix up the MX23 in upstream u-boot, it's just that some help other than Otavio might be needed. I don't have time to hack on it myself now :-(

- And if I would keep using the bootlets option from Freescale, I guess the currently elftosb2 tool generated sd_mmc_bootstream.raw can't be used as is in SPI flash ? Or maybe I'm still confused by something here, and I should study this area more.

Just generate the bootstream with elftosb2, elftosb -zf mx28 -c file.bd -o output.sb and write output.sb to your SPI flash.

(ie. usb loader fox mx23)

The USB loader was actually initially written for MX28, I added MX23 support only afterwards.

How do I specify "partitions" in SPI flash

You want to study mtdparts in u-boot and related biz. Look for "command line partition parsing" in Linux, it's like passing partitions to any other mtd device. You can also pass mtd partitioning scheme via DT in Linux, just check arch/arm/boot/dts/imx28-m28evk.dts in latest 3.7, it's right at the begining (it's there for GPMI NAND, but it works the same for SPI flash)

[jlumme]

Hi Marek,

Sorry for my late reply, I was a bit busy with my normal work, and couldn't work on my project.

Indeed, creating partitions was easy, as you said. Thank you!

I'm sorry to hear that imx23 work on uboot has been a bit slower recently..

I appreciate your advice about how to create a bootstream as well, it doesn't seem much different from what is currently being used in the makefile I have been using (elftosb2 -d -z -c ./linux_prebuilt.db -o i$(ARCH)_linux.sb).

I will try it immediately once I can see that SPI works ok..

I have been trying to stabilize SPI flash on linux side now, I seem to have run into a strange problem. For some reason when I just try reading raw data from mtd0, if I read more than 35 bytes, I get DMA errors.

If I read less than 35 bytes, it's always ok, but 36 or more, it goes haywire..

May I ask if you have you seen something like this before ? Reading from SPI flash more than 35 bytes causes driver timeout and results to corrupt - Electrical ...

[MarekVasut]

For some reason when I just try reading raw data from mtd0, if I read more than 35 bytes, I get DMA errors.

Does this happen with 3.7 mainline kernel? I know I fixed quite a lot of bugs in the original FSL SPI driver, which were very ugly.

I had similar issues where I ran into ugly DMA overruns when using the mtd interface. When I was reading too much data (over 32k), the DMA driver crashed ; this is fixed now.

Do not _ever_ directly write the /dev/mtdX , it will not work, use mtdutils for that. What will happen if you do is the MTD interface will issue the SPI flash erase command over the whole region you're planning to write, then SPI flash write command for the first block and then it will stream the whole block of data there. But the SPI flash needs one write command per block, thus only first block will be written.

My suggestion to you is to run the flash at 1MHz (very slow) and see if you can read it whole. Then try the /dev/mtdblockX interface and see if you can operate it via this interface. Then see if you can read the whole flash via /dev/mtdX (dd if=/dev/mtdX bs=4k count=N | sha1sum ... verify if the checksum matches by running it a few times, cycle the board , retry etc)

I don't expect this one will help, but it should kick you in the right direction. And ... well, sorry for the late reply ;-)

[jlumme]

Marek,

As always, thanks for your reply.

I'm not trying to read anywhere close to 32k, only 35 bytes, and I get screwed there.. O_o

I didn't know about the necessity of writing via mtdtools - I will need to look into how they work, thanks!

When slowing down the flash, is it correct if I limit the speed in the dtsi ? I lowered it to 10MHz yesterday, and at least the driver reported that, but I forgot to measure it with oscilloscope. I guess I should directly see it with the clock line..

dd is good idea, I will try today.

About the driver and kernel, this might be a noobish question; I currently take cloned the kernel from Freescale/linux-mainline · GitHub using: -b patches 3.7, does that make my kernel.. umm mainline kernel ?

[MarekVasut]

When slowing down the flash, is it correct if I limit the speed in the dtsi ?

Yes, that's the way :smileyhappy: Looking at it with a decent scope is a good idea, but you might hold onto something, you might be surprised by what you might see ;-)

About the driver and kernel

Just download the latest 3.7.1 from http://kernel.org or do:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

cd linux-2.6

git checkout -t -b my-3.7-branch v3.7

[jlumme]

Marek, I downloaded 3.7.1 tarball, and compared spi-mxs.c there with the one I'm using in Freescale's 3.7 branch.. the files are identical... :smileyconfused:

I wonder if my code with just reading a file pointer to /dev/mtd0 is to blame here or what... though I guess it should be more or less the same with dd. It will be interesting to see.

[MarekVasut]

Heh, so FSL pulled the upstream changes. That is good !

And if you're having issues there, it means something is broken and needs further investigation. Can you try switching the driver to PIO-only operation? How to do it -- easiest way is to find mxs_spi_transfer_one() function in drivers/spi/spi-mxs.c . Then locate condition if (t->len < 32) (there is a large block comment right above it) and replace it with if (1) . Such adjustment will assure the mxs_spi_txrx_pio() is always used to transfer data and thus DMA is never used.

Hope it helps, let me know the result ;-)

[jlumme]

Marek, indeed there seems to be some problem with DMA reads.

With PIO all is ok. I made a fancy table with different speeds, but it's not really needed. With PIO mode, all speeds seem OK.. I'm yet to verify full flash reads, but it operates so much like one would assume it would, that I'm pretty sure it's ok.

If you wish, and you have need for it, I can offer to try to help you debug this.. (ie. testing development drivers etc)

[MarekVasut]

Can you check all places in in mxs-spi.c , in the function mxs_spi_txrx_dma() that take into account IMX23_SSP ? There are two. Check if in the first one, the min variable doesn't overflow (see if it's no bigger than 0xffff, but it shouldn't be).

[jlumme]

Hi Marek,

I can't see any strangeness in how min value is selected (and len decremented), maybe I have misunderstood something. It seems to be chosen correctly; if remaining data to be read is bigger than SG_MAXLEN (0xff00), it's set as SG_MAXLEN, and last block will be set correctly to remaining data.

I did find something peculiar, though.

If reading only 1 DMA block (=reading less than SG_MAXLEN) the request will always timeout, if the amount of requested data is divided by 4.

So reading 32 will timeout, 33, 34 or 35 bytes, will be ok. 36 fail, 37 38 39 ok. This is valid "whole first block". Reading 65276 bytes will fail, but 65277, 78 & 79 will be ok.

When preparing more than 1 block, all reads fail, no matter the size.

You can see how min acts, and the results from above (shortened) in here http://paste.ubuntu.com/1456253/

I hope this helps you at least somewhat, please let me know if I can help you more..

[MarekVasut · Accepted solution]

I have this idea, try the following patch please:

diff --git a/drivers/spi/spi-mxs.c b/drivers/spi/spi-mxs.c

index eecefbc..ec80c46 100644

--- a/drivers/spi/spi-mxs.c

+++ b/drivers/spi/spi-mxs.c

@@ -243,6 +243,7 @@ static int mxs_spi_txrx_dma(struct mxs_spi *spi, int cs,

        INIT_COMPLETION(spi->c);

        ctrl0 = readl(ssp->base + HW_SSP_CTRL0);

+       ctrl0 &= ~BM_SSP_CTRL0_XFER_COUNT;

        ctrl0 |= BM_SSP_CTRL0_DATA_XFER | mxs_spi_cs_to_reg(cs);

        if (*first)

[jlumme]

Marek, thanks for your suggestion, indeed the bytes in CTRL0 needed zeroing. Now it's working!

[MarekVasut]

Please try the following adjusted patch to fix the DMA chaining issue:

diff --git a/drivers/spi/spi-mxs.c b/drivers/spi/spi-mxs.c

index eecefbc..7bb0b78 100644

--- a/drivers/spi/spi-mxs.c

+++ b/drivers/spi/spi-mxs.c

@@ -243,6 +243,7 @@ static int mxs_spi_txrx_dma(struct mxs_spi *spi, int cs,

        INIT_COMPLETION(spi->c);

        ctrl0 = readl(ssp->base + HW_SSP_CTRL0);

+       ctrl0 &= ~BM_SSP_CTRL0_XFER_COUNT;

        ctrl0 |= BM_SSP_CTRL0_DATA_XFER | mxs_spi_cs_to_reg(cs);

        if (*first)

@@ -258,8 +259,10 @@ static int mxs_spi_txrx_dma(struct mxs_spi *spi, int cs,

                if ((sg_count + 1 == sgs) && *last)

                        ctrl0 |= BM_SSP_CTRL0_IGNORE_CRC;

-               if (ssp->devid == IMX23_SSP)

+               if (ssp->devid == IMX23_SSP) {

+                       ctrl0 &= ~BM_SSP_CTRL0_XFER_COUNT;

                        ctrl0 |= min;

+               }

                dma_xfer[sg_count].pio[0] = ctrl0;

                dma_xfer[sg_count].pio[3] = min;