P1013 eTSEC data not DMA'ing

stevensnyder · ‎05-13-2016

I am attempting to use eTSEC1 as a network device but see no evidence that the packet is being DMA'd from the TX ring buffer. A scope on the transmit line between the P1013 and my PHY ( a TI DP83848) does not see TX asserted, and enabling loopback in MACCFG1 does not result in my packet appearing in a RX buffer.

The above is icing on the cake. The real indication of nothing happening is that the TX Ready bit in the TX data buffer descriptor is never cleared. There is no indication of error in IEVENT, or in TSTAT. It seems like the P1013 isn't snooping my memory accesses, although I have both RX and TX snooping enabled.

I an not using interrupts, just relying on the status bits in the eTSEC-related registers to determine state.

I don't know if this is relevant, but I am attempting to do RMII transfers. My PHY is configured (via MDIO) for Auto-Neg, supporting 10BASE-T and 100BASE-T, in both full- and half-duplex. I've tried both full- and half-duplex in MACCFG2 without success.

TX & RX descriptors and buffers are 16-byte-aligned in memory (shown in regs below), but even if they weren't wouldn't there be some indication of error?

The code below is the simplest test case I can come up with, and below that are the register values after the failure to transmit my packet.

Any thoughts as to why I seem unable to transmit from eTSEC1?

Thanks.

#define DMACTRL_TDSEN   0x01000080
#define DMACTRL_TBDSEN  0x02000040
#define DMACTRL_GRS     0x00000010
#define DMACTRL_GTS     0x00000008
#define DMACTRL_TOD     0x00000004
#define DMACTRL_WOP     0x00000001
#define TXBD_READY      0x8000
#define TXBD_LAST       0x0800
#define INIT_ECNTRL (ECNTRL_CLRCNT | ECNTRL_STEN | ECNTRL_R100M | ECNTRL_RMM)
#define INIT_MACCFG1 (MACCFG1_RX_EN | MACCFG1_TX_EN)
#define INIT_MACCFG2 (MACCFG2_IF_MODE_NIBBLE | MACCFG2_FULL_DUPLEX)
#define INIT_PHY100FD (BMCR_FULLDPLX | BMCR_SPEED100)
#define INIT_PHYRMII ( (1 << 5) | (1 << 0) )
#define INIT_DMACTRL (DMACTRL_TDSEN|DMACTRL_TBDSEN|DMACTRL_TOD|DMACTRL_WOP)
void simple_gpp_xmit(void) {
   // Per doc: 15.9.3.1.2 User initialization; page 976
   unsigned miistatus;
   int i, timeout=1000, txfailed=0;
   tsec_t         * const tregs = tsec_info[0].regs;
   tsec_mii_mng_t * const mregs = tsec_info[0].miiregs_sgmii;
   ccsr_gur_t     * const gregs = (unsigned int *)(CCSRBASE + 0xE0000);
   static const unsigned char packet[] = {
      0x00, 0x1e, 0x4f, 0x12, 0xcb, 0x2c, 0x00, 0x25,
      0x64, 0xbb, 0xd1, 0xab, 0x08, 0x00, 0x45, 0x00,
      0x00, 0x5c, 0xdd, 0x22, 0x00, 0x00, 0x80, 0x01,
      0x1f, 0x71, 0x0a, 0xc1, 0x14, 0x22, 0x0a, 0xc1,
      0x14, 0x6a, 0x08, 0x00, 0xef, 0x7e, 0x02, 0x00,
      0x94, 0x05, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66,
      0x67, 0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e,
      0x6f, 0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76,
      0x77, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67,
      0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f,
      0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77,
      0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68,
      0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, 0x70,
      0x71, 0x72};
   // 0. disable active DMA (just in case)
   out_be32(&tregs->dmactrl, in_be32(&tregs->dmactrl) | (DMACTRL_GTS|DMACTRL_GRS));
   // 1. Set and clear MACCFG1 [Soft_Reset]
   out_be32(&tregs->maccfg1, MACCFG1_SOFT_RST);
   udelay(2);
   out_be32(&tregs->maccfg1, INIT_MACCFG1 | MACCFG1_LOOPBACK); 
   // 2. Initialize MACCFG2
   out_be32(&tregs->maccfg2, INIT_MACCFG2);  // RMII 10/100 full-duplex
   out_be32(&tregs->ecntrl,  INIT_ECNTRL);
   // 3. Initialize MAC station address
   out_be32((&tregs->macstnaddr2), 0x60020000);
   out_be32((&tregs->macstnaddr1), 0x4365878C);
   // 4. Set up the PHY using the MII management interface
   // 4a : phy: reset
   out_be32((&mregs->miimadd), PHY_PHYS_MASK | 0x00);
   out_be32((&mregs->miimcon), BMCR_RESET);
   while ( in_be32(&mregs->miimind) & MIIMIND_BUSY ) { udelay(1); }
   // 4b : phy: wait until reset completed
   do {
      out_be32((&mregs->miimcom), MIIMCOM_WRIT_CYCLE);
      out_be32((&mregs->miimcom), MIIMCOM_READ_CYCLE);
      while (in_be32(&mregs->miimind) & MIIMIND_BUSY) { udelay(1); }
      miistatus = in_be32(&mregs->miimstat);
   } while (miistatus & BMCR_RESET);
   // 4c : phy: set fixed 100Mbs full-duplex
   out_be32((&mregs->miimadd), PHY_PHYS_MASK | 0x00);
   out_be32((&mregs->miimcon), INIT_PHY100FD);
   while ( in_be32(&mregs->miimind) & MIIMIND_BUSY ) { udelay(1); }
   // 4d : phy: set RMII
   out_be32((&mregs->miimadd), PHY_PHYS_MASK | 0x17);
   out_be32((&mregs->miimcon), INIT_PHYRMII);
   while ( in_be32(&mregs->miimind) & MIIMIND_BUSY ) { udelay(1); }
   // 5. Clear IEVENTGg
   out_be32(&tregs->ievent, IEVENT_CLEAR_ALL); // all bits set
   // 6. Initialize IMASKGg
   out_be32(&tregs->imask, IMASK_MASK_ALL); // all bits cleared
   // 7. Initialize RCTRL
   out_be32(&tregs->rctrl, RCTRL_BC_MPROM);  // promiscuous mode
   // 8. Initialize DMACTRL
   // 8a : init aligned data dma descriptors
   for (i = 0; i < RXQUEUELEN; i++) {
      rtx.rxbd[i].status = 0x8000;
      rtx.rxbd[i].length = 0;
      rtx.rxbd[i].bufPtr = (uint)NetRxPackets[i];
   }
   for (i = 0; i < TXQUEUELEN; i++) {
      rtx.txbd[i].status = 0;
      rtx.txbd[i].length = 0;
      rtx.txbd[i].bufPtr = 0;
   }
   rtx.rxbd[RXQUEUELEN - 1].status |= 0x2000;
   rtx.txbd[TXQUEUELEN - 1].status |= 0x2000;
   // 8b : set pointers to TX/RX descriptors
   out_be32(&tregs->tbase, (unsigned int)(&rtx.txbd[0]));
   out_be32(&tregs->rbase, (unsigned int)(&rtx.rxbd[0]));
   // 8c : enable TX data/BD snooping, polled transmit
   out_be32(&tregs->dmactrl, in_be32(&tregs->dmactrl) | INIT_DMACTRL);
   // 8d : clear GTS/GRS
   out_be32(&tregs->dmactrl, in_be32(&tregs->dmactrl) & ~(DMACTRL_GTS|DMACTRL_GRS));
   //
   // "transmit" a packet
   //
   // populate TX descriptor
   rtx.txbd[0].bufPtr = (unsigned int)&packet[0];
   rtx.txbd[0].length = sizeof(packet);
   rtx.txbd[0].status = (TXBD_READY | TXBD_LAST);
   // tell the DMA to go
   out_be32(&tregs->tstat, TSTAT_CLEAR_THALT);
   // wait for buffer to be transmitted
   for (i = 0; (rtx.txbd[0].status & TXBD_READY) && !txfailed; i++) {
      if (timeout--)
         udelay(1);
      else
         txfailed = 1;
   }
   if (txfailed) {
      // 15.9.3.14 Error-handling procedure; page 996
      unsigned short phyregs[0x20];
      printf("failed to xmit packet; TxBD[0] status: 0x%04x\n", rtx.txbd[0].status);
      // TSEC regs
      printf("maccfg1   == 0x%08x\n", in_be32(&tregs->maccfg1) );
      printf("maccfg2   == 0x%08x\n", in_be32(&tregs->maccfg2) );
      printf("ecntrl    == 0x%08x\n", in_be32(&tregs->ecntrl)  );
      printf("ievent    == 0x%08x\n", in_be32(&tregs->ievent)  );
      printf("imask     == 0x%08x\n", in_be32(&tregs->imask)   );
      printf("tctrl     == 0x%08x\n", in_be32(&tregs->tctrl)   );
      printf("rctrl     == 0x%08x\n", in_be32(&tregs->rctrl)   );
      printf("tbase     == 0x%08x\n", in_be32(&tregs->tbase)   );
      printf("rbase     == 0x%08x\n", in_be32(&tregs->rbase)   );
      printf("dmactrl   == 0x%08x\n", in_be32(&tregs->dmactrl) );
      printf("tstat     == 0x%08x\n", in_be32(&tregs->tstat)   );
      printf("rstat     == 0x%08x\n", in_be32(&tregs->rstat)   );
      // RMON regs
      printf("rmon tbyt == 0x%08x\n", in_be32(&tregs->rmon.tbyt)); 
      printf("rmon tpkt == 0x%08x\n", in_be32(&tregs->rmon.tpkt)); 
      printf("rmon tmca == 0x%08x\n", in_be32(&tregs->rmon.tmca)); 
      printf("rmon tbca == 0x%08x\n", in_be32(&tregs->rmon.tbca)); 
      printf("rmon txpf == 0x%08x\n", in_be32(&tregs->rmon.txpf)); 
      printf("rmon tdfr == 0x%08x\n", in_be32(&tregs->rmon.tdfr)); 
      printf("rmon tedf == 0x%08x\n", in_be32(&tregs->rmon.tedf)); 
      printf("rmon tscl == 0x%08x\n", in_be32(&tregs->rmon.tscl)); 
      printf("rmon tmcl == 0x%08x\n", in_be32(&tregs->rmon.tmcl)); 
      printf("rmon tlcl == 0x%08x\n", in_be32(&tregs->rmon.tlcl)); 
      printf("rmon txcl == 0x%08x\n", in_be32(&tregs->rmon.txcl)); 
      printf("rmon tncl == 0x%08x\n", in_be32(&tregs->rmon.tncl)); 
      printf("rmon res2 == 0x%08x\n", in_be32(&tregs->rmon.res2)); 
      printf("rmon tdrp == 0x%08x\n", in_be32(&tregs->rmon.tdrp)); 
      printf("rmon tjbr == 0x%08x\n", in_be32(&tregs->rmon.tjbr)); 
      printf("rmon tfcs == 0x%08x\n", in_be32(&tregs->rmon.tfcs)); 
      printf("rmon txcf == 0x%08x\n", in_be32(&tregs->rmon.txcf)); 
      printf("rmon tovr == 0x%08x\n", in_be32(&tregs->rmon.tovr)); 
      printf("rmon tund == 0x%08x\n", in_be32(&tregs->rmon.tund)); 
      printf("rmon tfrg == 0x%08x\n", in_be32(&tregs->rmon.tfrg)); 
      // misc P1013 regs
      printf("pordevsr  == 0x%08x\n", in_be32(&gregs->pordevsr)); 
      // PHY regs
      for (i=0; i<0x20; i++) {
         out_be32((&mregs->miimadd), PHY_PHYS_MASK | i);
         out_be32((&mregs->miimcom), MIIMCOM_WRIT_CYCLE);
         out_be32((&mregs->miimcom), MIIMCOM_READ_CYCLE);
         while (in_be32(&mregs->miimind) & MIIMIND_BUSY) { udelay(1); }
         phyregs[i] = in_be32(&mregs->miimstat);
      }
      printf("PHY[0x00]: 0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x\n",
             phyregs[ 0],phyregs[ 1],phyregs[ 2],phyregs[ 3],phyregs[ 4],phyregs[ 5],phyregs[ 6],phyregs[ 7]);
      printf("PHY[0x08]: 0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x\n",
             phyregs[ 8],phyregs[ 9],phyregs[10],phyregs[11],phyregs[12],phyregs[13],phyregs[14],phyregs[15]);
      printf("PHY[0x10]: 0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x\n",
             phyregs[16],phyregs[17],phyregs[18],phyregs[19],phyregs[20],phyregs[21],phyregs[22],phyregs[23]);
      printf("PHY[0x18]: 0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x,0x%04x\n",
             phyregs[24],phyregs[25],phyregs[26],phyregs[27],phyregs[28],phyregs[29],phyregs[30],phyregs[31]);
   }
}

failed to xmit packet; TxBD[0] status: 0x8800
maccfg1   == 0x00000105
maccfg2   == 0x00000101
ecntrl    == 0x0000100c
ievent    == 0x00000000
imask     == 0x00000000
tctrl     == 0x00000000
rctrl     == 0x00000008
tbase     == 0x00011800
rbase     == 0x00011810
dmactrl   == 0x000000c1
tstat     == 0x00000000
rstat     == 0x00000000
rmon tbyt == 0x00000000
rmon tpkt == 0x00000000
rmon tmca == 0x00000000
rmon tbca == 0x00000000
rmon txpf == 0x00000000
rmon tdfr == 0x00000000
rmon tedf == 0x00000000
rmon tscl == 0x00000000
rmon tmcl == 0x00000000
rmon tlcl == 0x00000000
rmon txcl == 0x00000000
rmon tncl == 0x00000000
rmon res2 == 0x00000000
rmon tdrp == 0x00000000
rmon tjbr == 0x00000000
rmon tfcs == 0x00000000
rmon txcf == 0x00000000
rmon tovr == 0x00000000
rmon tund == 0x00000000
rmon tfrg == 0x00000000
pordevsr == 0x19000040
PHY[0x00]: 0x2100,0x7849,0x2000,0x5c90,0x0101,0x0000,0x0004,0x2001
PHY[0x08]: 0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000
PHY[0x10]: 0x0005,0x0000,0x2000,0x0000,0x0000,0x0000,0x0100,0x0021
PHY[0x18]: 0x0000,0x8027,0x0904,0x0000,0x0000,0x6011,0x08be,0x0000

zwilcox · ‎12-18-2018

Again, I know this is a dated post. However, I found this question through searching the forums.

I'm going to post my solution just in case some one in the future has the same problem and is looking for a solution.

TBASE/RBASE must be 64 bit/16 byte aligned. TBASE/RBASE will truncate the the last 3 bits regardless of what you write.

I resolved this by adding an align 64:

zwilcox · ‎12-18-2018

I know this is a dated post. But were you able to resolve the issue with the TSEC?
This code looks similar to what's in UBOOT. I've done the same and having the same issue you are.

alexander_yakov · ‎05-13-2016

The most probable reason for this behavior is cacheable memory used for TX buffer descriptors. In this case memory coherency error happens - the core stores TX buffer descriptor contents to cache, but TSEC reads previous (unmodified) data from physical memory.

stevensnyder · ‎05-16-2016

Hi, Alexander.

Can you suggest a way to test this theory? I haven't done any explicit configuration of the ECM, so it has default values.

billpaul · ‎05-16-2016

You can try to perform a cache flush of the descriptor memory after you update it. This can be done with a dcbf instruction -- I don't know if you have a utility function set up for this already. If you do "dcbf <addr>" that will flush the cache line starting at addr. I don't remember what the cache line size is for the P1031, but it's larger than the 8 byte descriptor size. This makes it hard to flush just one descriptor, but as an experiment you can overlook this for now. In my drivers I usually allocate the descriptors in uncached memory, using the MMU to set the cache inhibited page attribute.

That said, the TSEC and eTSEC are supposed to support bus snooping for both data buffers and descriptors. If you enable snooping, manual cache flushing becomes unnecessary. (The hardware maintains cache coherency for you.) For the RX descriptors and buffers, you need to set bits in the ATTR register (RDSEN, RBDSEN). For the TX descriptors and buffers, you need to set bits in the DMACTRL register (TDSEN, TBDSEN). In your sample code, it looks like you're already setting the TDSEN and TBDSEN bits in DMACTRL, so this may not be the problem.

Also, I notice that when you initialize the TX descriptor, you set only the READY and LAST bits, You probably also want to set the INTERRUPT bit (0x1000). If you read table 15-834 in the P1022 reference manual (which also covers the P1031), you see that you have to set this bit in order for TX events to be reported in the IEVENT register. This won't actually cause an interrupt to occur unless you also unmask the corresponding bits in the IMASK register.

Lastly, please indicate that you have considered what I said about enabling the receiver and transmitter last and clearing the GTS and GRS bits in the DMACTRL register as the last step in the controller initialization. You should have the DMA ring pointers initialized before you enable DMA and start the receiver and transmitter running.

billpaul · ‎05-13-2016

There's some information that's not clear from your question. You have shown us your code, but you haven't told us what environment it's running in. Is this inside the Linux kernel? Your own custom OS? Is it just a standalone module? Is this a Freescale reference board? is it your own custom designed board? Where is tsec_info populated?

There's a couple of odd things I see:

1) On line 49 you're setting the RX and TX enable bits in MACCFG1 right away. I usually wait to enable them (and the bits in the DMACTRL register) last, after I've already set up the RX and TX DMA ring pointers.

2) Your PHY register dump shows the status register has value 0x7849. This means there's no link present. Note however that the link state bit is a latching bit, so you have to read it twice to get the real time current link state. It could be the link is up but you need to read the registers again to see it.

3) Do you have the MMU turned on? Are your DMA ring and buffer addresses virtual or physical?

If you're not using a 1:1 virtual to physical mapping, then you have to do a virtual to physical address translation on your DMA ring pointers and buffer addresses, otherwise you'll be telling the eTSEC to use the wrong source/destination addresses for DMA transfers.

stevensnyder · ‎05-16-2016

Hi, Bill.

This is bare-metal code; any memory addresses referenced are physical DDR. I do not have the MMU turned on.

Regarding the PHY link status: yes, I saw that, but also don't see any evidence that the transmit is even attempted. Even if the link is down, shouldn't I see at least come activity in the counters, or some error in TSTAT? As it is there is just nothing.

Also, I am assuming that with loopback set in maccfg1 the P1013 shouldn't care about the state of the PHY. I could be wrong about that, but the loopback should limit the actions to DMA'ing the packet from the TX BD into the chip, then back out to the RX BD.