topic Re: P5020 Multibit ECC in P-Series

P5020 Multibit ECC

timothypark — Thu, 09 Mar 2017 23:27:54 GMT

Hello,

I am trying to test the multi-bit ECC capability on the P5020, however, I seem to be running into an issue where I hit multiple machine check errors after the first multi-bit ecc is handled. I have the following set up to inject the multi-bit ecc and also to handle the machine check exception to follow:

DDR_ERR_DISABLE = 0

DDR_SDRAM_CFG[ECC_EN] = 1

DDR_ERR_INJECT_LO = 0x00000003

DDR_ERR_INJECT[EIEN] = 1

IVOR1 is set to handle the machine check exception by rolling up MCSRR0 to the next instruction once the rfmci instruction is called.

After the first multi-bit ecc is handled the injection is turned off as such: DDR_ERR_INJECT[EIEN] = 0.

It is after this where every instruction step will trigger a multi-bit error and it keeps vectoring off to IVOR1.

Please let me know what else I should be doing.

Thanks,

Tim

Re: P5020 Multibit ECC

Bulat — Fri, 10 Mar 2017 12:09:31 GMT

I believe your test proves that multi-bit ecc errors are detected by the memory controller. Or the purpose of your test is different?

Regards,

Bulat

Re: P5020 Multibit ECC

timothypark — Fri, 10 Mar 2017 15:55:10 GMT

Hello Bulat,

Thanks for your reply.

Well, yes, the machine check error does prove that the DDR controller detected the multi-bit ecc that I wanted. However, it is after turning off the ECC injector that my problem arises. I keep running into multi-bit ecc errors and vectoring off to IVOR1 after each instruction step. I know I keep running into multi-bit ecc, even after turning off the injector, because the DDR_ERR_DETECT[MBE] gets set every time I clear it with each instruction step.

Am I doing something wrong or am I miss understanding about multi-bit ecc and this is just the proper operation after a multi-bit ecc has been detected?

Thanks,

Tim

Re: P5020 Multibit ECC

Bulat — Mon, 13 Mar 2017 12:31:36 GMT

It is difficult to guess what is really happening, problem can relate to particular configuration (like cache on/off). Do you read 'DDR_CAPTURE_n' registers of the DDR controller in the interrupt handler? Do you reset error bits in the ERR_DETECT register?

Regards,

Bulat

Re: P5020 Multibit ECC

timothypark — Mon, 13 Mar 2017 17:28:08 GMT

During initialization the TLB for the DDR is set so that caching is inhibited, so I do not believe it to be a cache issue. Unless there is something else I have to set to disallow DDR to be cached.

Within the interrupt handler, reading and clearing the DDR_CAPTURE_n registers doesn't seem to help. I do reset the ERR_DETECT register within the interrupt handler so that it is clear when it returns back to normal code. However, it get set again with the new unexpected multi-bit error. I should also mention that the DDR_CAPTURE_n does get set to the new multi-bit error.

Thanks,

Tim

Re: P5020 Multibit ECC

Bulat — Tue, 14 Mar 2017 16:41:14 GMT

So you can see a lot of different addresses reported via DDR_CAPTURE_ADDRESS. Do these address values look reasonable?

Are interrupts enabled in the DDRx_ERR_INT_EN register? If yes, try to disable it.

Regards,

Bulat

Re: P5020 Multibit ECC

timothypark — Thu, 16 Mar 2017 21:14:52 GMT

Sorry for the late reply, was busy with another task for a bit. I believe that the address within the DDR_CAPTURE_ADDRESS to be reasonable, but I will have to confirm with you if there is something odd. All I would see is that the address captured there were similar to the address of the instruction location.

I've tried multiple of different settings, including disabling DDRx_ERR_INT_EN for multi-bit (i.e. MBEE is set to 0) and also enabling it. All of this and still get the same problem of multi-bit error after turning off the injector.

Thank you for taking the time helping me investigate this.

Tim

Re: P5020 Multibit ECC

Bulat — Fri, 17 Mar 2017 17:44:18 GMT

I can not know what happens on your board. However something really looks odd.

You mentioned following setting describing test setup:

DDR_SDRAM_CFG[ECC_EN] = 1

Is that done in the test or during memory initialization?

Can you provide values of all DDR_CAPTURE registers and MCSRR0 during first MCE?

The same information during second MCE?

Regards,

Bulat

Re: P5020 Multibit ECC

timothypark — Tue, 21 Mar 2017 00:04:41 GMT

Yes, the board is acting odd. I'm sure it is something that I haven't or have set up wrong, but I'm unsure what it is. Thanks again for working with me.

DDR_SDRAM_CFG[ECC_EN] = 1 is done during memory initialization, we are using Code warrior and its TCL script to set up the board for our testing.

Here is the registers before:

CAPTURE_DATA_HI: 0x919a0e08

CAPTURE_DATA_LO: 0x3b20fffc

CAPTURE_ECC: 0x49494949

CAPTURE_ATTRIBUTES: 0x10802001

CAPTURE_ADDRESS: 0x000050d0

MCSRR0: 0x000050d4

MCSRR1: 0x2000

Here is the registers after:

CAPTURE_DATA_HI: 0x919a0e08

CAPTURE_DATA_LO: 0x3b20fffc

CAPTURE_ECC: 0x49494949

CAPTURE_ATTRIBUTES: 0x12492001

CAPTURE_ADDRESS: 0x000050d0

MCSRR0: 0x000050d8

MCSRR1: 0x2000

Thanks,

Tim

Re: P5020 Multibit ECC

Bulat — Tue, 21 Mar 2017 12:28:46 GMT

Can you take memory dump that includes address 0x000050d0 before errors are injected? How different the data in the memory and in the CAPTURE_DATA registers? Does this correspond to the DDR_ERR_INJECT_LO = 0x00000003 mask?

Also it is not clear how the code can be affected by the error injection... What are you doing exactly after DDR_ERR_INJECT[EIEN] = 1?

Regards,

Bulat

Re: P5020 Multibit ECC

timothypark — Tue, 21 Mar 2017 18:51:12 GMT

So interesting things happen when I do the memory dump before and after the first MCE. Here is the memory dump before the first MCE:

0x000050c0: 0x7d9cd830 0x81990e04 0x7d8ce378 0x91990e04

0x000050d0: 0x81990e08 0x618c0100 0x91990e08 0x3980ffff

0x000050e0: 0x919f0000 0x81990e08 0x558c062c 0x91990e08

Here is the memory dump after coming back from the first MCE

0x000050c0: 0x7d9cd830 0x81990e04 0x7d8ce378 0x91990e04

0x000050d0: 0x7d9cd830 0x81990e04 0x7d8ce378 0x91990e04

0x000050e0: 0x7d9cd830 0x81990e04 0x7d8ce378 0x91990e04

As you can see, addresses from 0x50d0 - 0x50ec get changed to almost match what is written in address ranges 0x50c0 - 0x50cc. These data that are located at these address ranges are the instructions for the code that tests multi-bit ECC.

They get changed just as I do a write of data into global data, which in this case the injector should kick in and alter the ECC for that global data.

Here are the steps just after enabling injection (i.e. DDR_ERR_INJECT[EIEN] = 1):

1) Write of 0xFFFFFFFF to a global variable

2) Read from global variable to a local variable

3) Disable error injection.

Thanks,

Tim

Re: P5020 Multibit ECC

Bulat — Wed, 22 Mar 2017 12:15:33 GMT

The chaos around instructions does not look like effect of the error injection.

Can you slightly change your sequence like following:

1) Write of 0xFFFFFFFF to a global variable

2) Disable error injection.

3) Read DDR_ERR_INJECT to be sure it is written

4) Read from global variable to a local variable

Regards,

Bulat

Re: P5020 Multibit ECC

timothypark — Wed, 22 Mar 2017 15:55:10 GMT

Still wonder why those instructions are being corrupted though when ECC is turned on.

I remember trying those steps in my attempt on figuring out multi-bit ECC and it not being fruitful, but I will try it again and get back to you.

Thanks,

Tim

Re: P5020 Multibit ECC

timothypark — Wed, 22 Mar 2017 19:25:50 GMT

Sadly, still getting the same result where the instructions are being corrupted after a MCE when following the steps you recommended.

Thanks,

Tim

Re: P5020 Multibit ECC

Bulat — Fri, 24 Mar 2017 12:27:23 GMT

Can you check HID0[EMCP] value before test starts?

Re: P5020 Multibit ECC

timothypark — Mon, 27 Mar 2017 15:21:33 GMT

It is set, HID0[EMCP] = 1. Also, MSR[ME] = 0 before test start too.

Re: P5020 Multibit ECC

Bulat — Wed, 29 Mar 2017 04:22:13 GMT

Is it correct: MSR[ME] = 0?

As I wrote, it is really difficult to guess what is happening, however it looks like your MCE events do not relate to ECC errors as it is supposed to be. I think you need to simplify your test to get positive result. This means following: as soon as you read a faulty word, the processor should immediately capture the address of the word in the CAPTURE_ADDRESS register. Not the address of the instruction that performs the read. I believe this can be done using step-by-step execution using a debugger.

Regards,

Bulat