e500mc tlbivax instruction bcast feature.

[vasanthsri]

Hi All,

Regarding the "MMU_FTR_USE_TLBIVAX_BCAST" MMU feature for e500mc.

Snippet from the e500mc manual (Table  3 -43) :

"

Thus, an invalidate operation is broadcast throughout the coherent domain of the

processor executing tlbivax"

Does this mean this core supports MMU_FTR_USE_TLBIVAX_BCAST feature ? if no, could someone provide me a clarification on this instruction behavior on e500mc ?

If yes, any idea why same is not added in the below e500mc CPU table code ?

Linux : 2.6.34

Source : powerpc/kernel/cputable.c

                .pvr_value              = 0x80230000,

                .cpu_name               = "e500mc",

                .cpu_features           = CPU_FTRS_E500MC,

                .cpu_user_features      = COMMON_USER_BOOKE | PPC_FEATURE_HAS_FPU,

                .mmu_features           = MMU_FTR_TYPE_FSL_E | MMU_FTR_BIG_PHYS |

                        MMU_FTR_USE_TLBILX,

                .icache_bsize           = 64,

                .dcache_bsize           = 64,

                .num_pmcs               = 4,

Thanks in Advance.

[lunminliang]

Hi,

I  think e500mc supports the TLB invalidation operation throughout the coherence domain.

When I read the code of Linux in Freescale SDK 1.6 which based on kernel 3.12

Source arch\powerpc\include\asm

/* Enable use of broadcast TLB invalidations. We don't always set it

* on processors that support it due to other constraints with the

* use of such invalidations

*/

#define MMU_FTR_USE_TLBIVAX_BCAST    ASM_CONST(0x00040000)

Could it be an explanation?

[vasanthsri]

Thanks luminliang for prompt on this query.

Yes, I did see that comment above the "MMU_FTR_USE_TLBIVAX_BCAST" macro , but just wondering when we have this feature is available on e500mc in linux code there are instances in which "mmu_has_feature(MMU_FTR_USE_TLBIVAX_BCAST)" hook is evaluated as "false"  (atleast in my version 2.6.34)  and this result (feature available ) is used to perform the broadcast using instruction (TLBIVAX) or (if not available) smp specific API is used to execute this operation on individual/masked/topology CPUs.

If there is any real constraints which is blocking this feature to be not available (software perspective) on e500mc , then would like to know those constraints to understand how -system wide- importance is required for those points.

[scottwood]

The hardware supports it for backwards compatibility reasons, but tlbilx is preferred as it avoids broadcasting to cores where the invalidation isn't needed (I don't know if any actual performance comparisons have been done), is easier/faster to virtualize, and doesn't require tlbsync (which requires software to provide mutual exclusion as multiple tlbsyncs at once can hang the system).  Plus, there have been errata with tlbivax/tlbsync.

[vasanthsri]

Scott Wood wrote:

The hardware supports it for backwards compatibility reasons, but tlbilx is preferred as it avoids broadcasting to cores where the invalidation isn't needed (I don't know if any actual performance comparisons have been done), is easier/faster to virtualize, and doesn't require tlbsync (which requires software to provide mutual exclusion as multiple tlbsyncs at once can hang the system).  Plus, there have been errata with tlbivax/tlbsync.

I re-read the section "6.4.4 TLB Invalidate Local Indexed (tlbilx) Instruction" after your comment and it all make sense now to me. Thanks again.I have three queries left before I go ahead with little custom changes to flush the TLB page.

- TLB broadcasting,

     - If the process (mm context) is expected not to migrate (but have clone running on other cores) to other CPU (like hard affined) then is it required to perform the broadcasting of flush operation ?

- TLB PID FLUSH

     - if we execute "_tlbil_pid" implementation , then the process mmap protection remains untouched ?

- TLB with _tlbivax_bcast

     - What would be the possible consequence if I force perform the _tlbivax_bcast instead of using smp_* function to perform the flush on all the CPU

Thanks in Advance.

[scottwood]

The TLB invalidation must be done on all CPUs that may have the translation in the TLB.  It's based on where the mm has been used on the past, not where it's going in the future.

I don't understand the second question.  _tlbil_pid() invalidates all mappings of the current PID on the current CPU.

If you want to use tlbivax_bcast, make sure you also enable MMU_FTR_LOCK_BCAST_INVAL, and check the errata document for the chip and revision you have.  Other than that, the main consequence is that you'd be the first one to use it on Freescale chips (even on e500v2, Linux doesn't use tlbivax -- it uses tlbsx+tlbwe to invalidate locally).  You'll need to supply a _tlbivax_bcast.  Neither of the current versions will be built on a 32-bit e500mc kernel, and even if you change the ifdef on the CONFIG_PPC_BOOK3E version, there may be IBM-specific assumptions in there -- in fact, I'm not sure any chip is using that version.  It seems that only 476 chips have MMU_FTR_USE_TLBIVAX_BCAST.  If you run into any other problems you'll need to debug.

[vasanthsri]

The TLB invalidation must be done on all CPUs that may have the translation in the TLB.  It's based on where the mm has been used on the past, not where it's going in the future.

Perfect. If we have a process which ensures its whole running lifetime (start-run-finish) is on one CPU then broadcast is not required with an assumption that mm is always going to be local to that CPU.

I don't understand the second question.  _tlbil_pid() invalidates all mappings of the current PID on the current CPU.

Maybe my understanding towards the _tlbil_pid() is wrong. Say,If we have a (userland) process which does mmap certain region (x ~ y) with custom protection settings (few segments as RO, rest as EXE). We modify a -bit- in pte then flush that page using flush_tlb_page which I believe as the standard way to ensure the changes are reflected properly on the local CPU. But, when the number of pages to be flushed increased instead of doing the per page flush, thought can drop(invalidate) whole process pages in one attempt using "_tlbil_pid" and ready to afford the "page faults" after this action. Here, I had question that, whether this per pid invalidation affects the protection (RO, EXE) and mmap (x ~y) of the process.

If you want to use tlbivax_bcast, make sure you also enable MMU_FTR_LOCK_BCAST_INVAL, and check the errata document for the chip and revision you have.  Other than that, the main consequence is that you'd be the first one to use it on Freescale chips (even on e500v2, Linux doesn't use tlbivax -- it uses tlbsx+tlbwe to invalidate locally).  You'll need to supply a _tlbivax_bcast.  Neither of the current versions will be built on a 32-bit e500mc kernel, and even if you change the ifdef on the CONFIG_PPC_BOOK3E version, there may be IBM-specific assumptions in there -- in fact, I'm not sure any chip is using that version.  It seems that only 476 chips have MMU_FTR_USE_TLBIVAX_BCAST.

Thanks. I'll attempt this and will post the result later next week. The most worry part is "system hang" which we can't afford incase it is reproduciable with this change (_tlbivax_bcast).

If you run into any other problems you'll need to debug.

:-)

[scottwood]

Invalidations don't change protection bits or any other aspect of translations.  They just remove potentially stale cached entries from the TLB, not the page table data itself.

[vasanthsri]

Scott Wood wrote:

Invalidations don't change protection bits or any other aspect of translations.  They just remove potentially stale cached entries from the TLB, not the page table data itself.

Thanks a bunch again.

I can see most of the invocation instances of "_tlbil_pid ()" is seems to be during process life cycle (startup/finish) , is this  API expected to get invoked only in those circumstances ?

If I invoke this against the "user land" process, will this one make all the entries stale/TLB pollution ?

[scottwood]

_tlbil_pid() can be called at any time.