SGI interferes with EPIT

[MikeJones]

I am having a problem with interrupts on an iMX6Q. EPIT interrupts seem to work until a SGI interrupt is sent. The SGI is handled, then EPIT interrupts stop.

The basic scenario is:

- Setup EPIT for CPU 0.

- Interrupts for EPIT are handled on CPU 0 only without any problems

- CPU 1 sends SGI to CPU 0, and EPIT interrupts stop

Here are the details from code using the SDK.

I have setup the EPIT from CPU 0 with the following:

void hal_clock_initialize(cyg_uint32 period)

{

    hal_epit_counter_enable(2, period, IRQ_MODE);

    _period = period;

}

The IRQ number is fetched with this code on CPU 0 when interrupt 89 occurs:

int get_IRQ(void)

{

    cyg_uint32 vector_num;

    cyg_uint32 cpu_num;

    cyg_uint32 irq_num;

    vector_num = gic_read_irq_ack();

    if (vector_num & 0x0200)

    {

        // Spurious interrupt.

        hal_gic_write_end_of_irq(vector_num);

        irq_num = INTERRUPT_NONE;

    }

    else

    {

        cpu_num = (vector_num >> 10) & 0x7;

        irq_num = vector_num & 0x1FF;

        if (irq_num < ISR_MIN || irq_num > ISR_MAX)

            irq_num = INTERRUPT_NONE;

    }

  return irq_num;

}

The interrupt is acknowledged on CPU 0 with the following code, which enables the next interrupt:

void hal_interrupt_acknowledge(int vector)

{

    cyg_uint32 irq_num;

    cyg_uint32 cpu_num;

    cpu_num = (vector >> 10) & 0x7;

    irq_num = vector & 0x1FF;

    if (irq_num != CYGNUM_HAL_INTERRUPT_NONE)

        gic_write_end_of_irq(irq_num);

}

On CPU 1, the following code issues the SGI to CPU 0:

        hal_gic_send_sgi(SMP_CPU_INTERRUPT_VECTOR(cpu), 1 << cpu, kGicSgiFilter_UseTargetList);

On CPU 1 the ISR is handled by calling get_IRQ then calling this:

__externC WORD32 cpu_message_isr( WORD32 vector, ADDRWORD data )

{

    SMP_CPU_TYPE me = SMP_CPU_THIS();

    WORD32 ret = 2;

    gic_enable_irq(vector, false);

    interrupt_acknowledge(vector);

    SPINLOCK_SPIN( m->lock );

Do Stuff

    SPINLOCK_CLEAR( m->lock );

    gic_enable_irq(vector, true);

    return ret; // Indicates to call DSR

}

This then calls a DSR:

__externC CYG_WORD32 cpu_message_dsr( CYG_WORD32 vector, CYG_ADDRWORD data )

{

    SMP_CPU_TYPE me = SMP_CPU_THIS();

    struct smp_msg_t *m = &smp_msg[me];

    CYG_WORD32 reschedule;

    gic_enable_irq(vector, false);

    SPINLOCK_SPIN( m->lock );

Do Stuff

    SPINLOCK_CLEAR( m->lock );

    gic_enable_irq(vector, true);

Do Stuff

}

The ISR and DSR do get called, and and once called, the EPIT interrupts no longer generate an interrupt on CPU 0. By breaking with the debugger, I can see the CPSR is in SVC mode and FIQ and IRQ are not masked on CPU 0. The EPIT is setup with priority 1. The SGI is setup with priority 0. Changing SGI priority to 2 makes no difference. SGIs from CPU 0 to CPU 1 happen more than once. SGIs from CPU 1 to CPU 0 stop EPIT interrupts.

From my logging I can see EPIT generate multiple interrupts, then right after the SGI from CPU 1 to CPU 0 is handled, all EPIT interrupts stop.

My understanding is that once the SGI is handled, it should no longer block any other interrupts. Perhaps there is some subtle issue with ordering, but I used the disable, ack, enable ordering in my UART driver and EPIT without any problems.

Any ideas what might stop the EPIT interrupts?

Any ideas on how to debug the problem?

[MikeJones]

I finally solved this after reading the ARM GIC Reference Manual a third time. I missed the fact that writing to GICC_EOIR requires the IRQ and the CPU. The GIC manual says that for SWI you must use CPU, otherwise it is SBZ. Because sometimes CPU is zero, it worked in some cases and not others, and this made it hard to understand what was going on. Furthermore, from an experiment, it appears that the iMX6 wants CPU for all cases. However, I am not 100% sure as I did not go back and recheck it. However, I can say with certainty it does not cause problems to use CPU for all cases.

I was also having problems with EPIT. I was trying to use IRQ priority 1 in GICD_IPRIORITYRs, and any time you set it to one, it reverts to 0 when you read it back. It appears that you have to start at 8 before the value persists. I have not gone back and tried to prove that 0 works and I could not find any documentation that gives the range of acceptable values. But when it is 8, my code works.

[MikeJones]

I have a little more information. At the time of the interrupts have stopped, the following are the case:

EPIT PENDING on CPU 0

SGI PENDING/ACTIVE CPU 0

My logging suggests that just before reaching this case, in order, the following occurred:

EPIT ISR/DSR processed on CPU 0

SGI ISR/DSR processed on CPU 0

SGI sent from CPU 0 to CPU 1

SGI ISR/DSR processed on CPU 1

SGI sent from CPU 1 to CPU 0

There is no log info to support getting the SGI on CPU 0 into the ACTIVE state. CPU 0 is in a loop spinning at this point and is alive.

There is no log info to support receiving the EPIT ISR on CPU 0.

The behavior is as if they were ignored.

All the GIC registers look ok. CPSR on CPU 0 is Hex:0x60000013.

I suppose the exception code could have failed to deliver the ISR, but that code works flawlessly for hours if the SGI is disabled.

If the EPIT was PENDING/ACTIVE, and then processed, will the PENDING status cause another exception? It seems that it must or you can never safely exit the exception without risk of missing a new PENDING.

Can the PENDING/ACTIVE on SGI at priority 0 block an interrupt from the EPIT at priority 1? That seems logical.

[MikeJones]

A little more information.

I checked that the distributor is enabled. Was ok.

I checked that the target is set. Was ok.

I checked the mask level in GICC_PMR. Was 248. Seems to be this after setting to 255.

I made a call to hal_gic_read_irq_ack() and get 1023. The arm doc says this is spurious on a gic with groups. All three items that can cause it were checked above.

So essentially, there are interrupts available, one SGI, one EPIT, and the distributor is ok, the target is ok, the mask level is ok, CPSR is ok, yet nothing is returned but spurious.

SGI is edge sensitive. EPIT is set to level sensitive. Changing EPIT to edge sensitive makes no difference.

I assume that if CPSR masks IRQ, and an interrupt then becomes pending, unmasking generates an IRQ. Therefore, code that is disabling IRQ with CSPR should not let stuff fall into a crack. Even if it did, I should not be getting a 1023 back.

[AnsonHuang]

Hi, Mike

    Sorry that I did NOT get through all your code and your case, but I did tried use EPIT before on i.MX6Q, and it works just fine, attached is the patch for enabling EPIT, you can have a refference.