Hello to you all! In this blog, we will focus on low power management in PCIe. PCIe Power management aims at reducing power consumption in PCIe devices by putting them to low power states when they are idle and bringing them back to full power state when required.
Why does it matter ? Because PCIe devices in the market such as graphic cards, storage devices, Network Interface Cards can consume significant power when they are active. Power management allows them to scale down their power consumption during idle or periods of low activity.
The scope of PCIe power management is comparatively huge and the readers are encouraged to go through the PCIe specification document in case they are looking for every possible detail.
We are going to touch upon some important concepts that are just enough to get started. So this blog will stick to the following agenda:-
- Introduction to PCIe power management
- What is PCIe LTSSM ?
- What are L1 sub-states and where do they fall inside the LTSSM?
- How does a PCIe device enters and exit the ASPM L1 Sub-states?
- How is ASPM exercised in linux kernel?
- Quick look at the PCIe analyzer traces
Introduction to PCIe power management
In the grand scheme of things, PCIe power management is exercised at 2 levels:-
a. Link
Link power management conserves power on the serial link within a PCIe fabric when there's no active data transfer. This can happen even when the device is fully powered on. ASPM[Active State Power Management] is responsible for managing the power consumption of this link. Link States such as L0,L0s[Standby], L1, L1ss[Sub-states], L2 and L3 fall under this category.
LTSSM
b. Device
Device power management aims at putting the entire device into low-power state when not in use. PCI-PM is responsible for managing the power states of the entire device. States such as D0,D1,D2.D3Hot and D3Cold fall under this category.
This blog will focus on the first category i.e Link Power Management using ASPM. We will take up PCI-PM in another blog.
What is PCIe LTSSM?
LTSSM acronym for Link Training and Status State Machine, is a finite state machine within the PCIe physical layer that ensures a link between two devices[Root Complex and Endpoint] is properly established and maintained.
It is responsible for managing the link initialization, link configuration, link recovery and power states among other things. Since it is a FSM, it operates through some defined states. It transitions between these states based on various conditions.
Some of the major states in the LTSSM:
- The device checks for the presence of another PCIe device.
- Devices exchange training sequences to align link parameters.
- Link width and speed are negotiated.
- Normal data transmission occurs.
- L0s / L1 / L2 (Low Power States)
- Power-saving modes when the link is idle or in standby.
- Used to recover from errors or renegotiate link parameters.
- Disabled / Hot Reset / Loopback
- Special states for testing, error handling, or reset.
The arrow in the above illustration denotes that one state can be transitioned to another under certain conditions. Please do not stress too much about every state in the diagram as this blog will only touch upon the ones that are needed to understand the L1 sub-states.
What are L1 sub-states and where do they fall inside the LTSSM?
To give you some context, L0 > L0s > L1 > L2 > L3 is the sequence of link states from the one that consumes most power[L0] to the one that consumes the least[L3]. It also means that the time to resume from low-power to active state increases in the reverse order. Meaning that the time taken by a PCIe device to go from L3 to L0 is greater than L1 to L0 transition.
In the previous section, we had mentioned the following link states:-
L0 - Active state in which data transmission happens.
L0s [Standby] - Link logically remains at L0 but physical signaling is reduced. No data transfer occurs in this state. Just minimal activity to keep the link going.
L1 - State providing greater power savings than L0s but with a cost of greater recovery latency. Also known as L1.0, It results in the deactivation of 'Link and Transaction' layer within each device.
To further reduce link power consumption, optionally L1 can have sub-states - L1.1 and L1.2. This also means that the recovery time latency increases from these low-power states to the active state.
L2 - In this state power can be aggressively conserved. Most of Transmitter and Receiver may be shut off. Main power and clock are not guaranteed but Auxiliary power is available. L2 support is dependent on the availability of Aux power on the board.
Note:- iMX95EVK does not support L2 state as it does not have an Auxiliary power.
L3 - When no power is present, the component is said to be in L3 state. This means that the link and device are completely inactive as there's no power.
Let's try to dig deeper into the L1 Sub-states of PCIe:-
Below is a state diagram that depicts the transitions into/out of L1 sub-states. It starts with L1.0 which is same as L1 explained previously. This also shows that in order to Enter L1.1/L1.2, the link state should go to L1.0 first and then these states may be entered. The link does not transition directly from L1.1 to L1.2 without entering L1.0 first.
L1.1
- In the L1.1 link state, the PLL,RX/TX circuits are OFF but Link common mode voltages are maintained to allow faster recovery from low-power state.
- Power savings are moderate.
- A bi-directional open-drain clock request signal i.e. CLKREQ# is used for entering and exit from this state.
- The Upstream and Downstream ports are not required to be enabled to detect Electrical Idle.
L1.2
- In the L1.2 link state, the PLL,RX/TX circuits are OFF and Link common mode voltages are not required to be maintained.
- Power savings are more than L1.1 but higher exit latency than L1.1
- A bi-directional open-drain clock request signal i.e. CLKREQ# is used for entering and exit from this state.
- The Upstream and Downstream ports are not required to be enabled to detect Electrical Idle.
From the above discussion on L1.1 and L1.2, it is important to understand the role of CLKREQ# signal in ASPM. In L1.2 for example, the reference clock is cut-off, so in order for the device to request the clock back, this signal is used. When the device wants to exit the L1.2 low power state, It asserts the CLKREQ# low.
Because CLKREQ# is a bi-directional open-drain signal, both host and device can potentially drive this signal.
How does a PCIe device enters and exit the ASPM L1 Sub-states?
The discussion in this section will assume that we have the below setup:-
Root-Complex End-point
iMX95EVK <--------> AW693 Wi-Fi module
M.2 Key.E slot
BSP used - 6.6.52 Linux Factory
The PCIe device[if supported] should advertise that it supports ASPM L1.1 or L1.2 by setting ASPM L1.1 and ASPM L1.2 bits. This setting of bits must be performed at the downstream port before it is done at the upstream port.
To determine if the PCIe device has ASPM enabled
The Link capability register specifies a device's support for ASPM. Moreover, software can enable and disable ASPM via 'Active State PM Control' field of the 'Link Control Register'. This is what Linux PCI drivers use to enable/disable ASPM.
On linux, you can execute "lspci -vvv" and check the output for your device. Example :-
In the snippet above, we observe that ASPM is enabled for this device.
Transition to low-power states
L0 is the active link state where data transfer happens. After the link has been trained and established, it is said to be in L0. The link enters and exits the L1 from/to L0 state.
The specification suggests that the condition for L0 to enter L1 is that both directions of the link have entered L0s and have been in this state for a preset amount of time.
After reaching L1.0, the controller is capable of automatically moving to deeper power-saving L1 sub-states i.e L1.1 and L1.2 can be entered. This happens in response to the de-assertion of the CLKREQ# signal, which indicates that the reference clock should be shut off.
To initiate or indicate transitions in these power management states, the DLLP layer send/receive Power Management messages. As part of the protocol, these messages are exchanged between Root-Complex and End-point to ensure that both the link partners are prepared for the transition. At the end of the message exchanges, the PCIe controller moves to a state machine[L1.0 or L2/L3] in which clock can be cut off.
We can understand this using the following scenario: -
*** Consider a message exchange required to Enter L1 ASPM Sub-states from L0 ***
The downstream component sends PM_Active_State_Request_L1 DLLP repeatedly to the Upstream component. After receiving this, the upstream component can either accept the request by sending PM_Request_Ack DLLP or reject by sending PM_Active_State_Nak TLP
Upon getting PM_Request_Ack DLLP, the downstream port stops sending the request DLLP and disables the DLLP & TLP transmission, places its transmit lanes into Electrical Idle state.
When the upstream component receives an Electrical Idle ordered set on its Receive Lanes, it stops sending the PM Request Ack DLLP, disables DLLP & TLP transmission, placing its transmit lanes into Electrical Idle state.
At this point, the link is in L1 state.
Further, if the PCIe device supports L1 sub-states and it is advertising the same in its link capabilities register, it can de-assert CLKREQ# signal to indicate that it does not need clock. The system then turns off the clock. The link goes to L1 substates.
How is ASPM exercised in linux kernel?
To configure PCIe ASPM in linux kernel, ensure that the following CONFIGS are enabled in the defconfig:-
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_POWER_SUPERSAVE=y
Build the kernel and then deploy it on the board.
At linux console, execute the following :-
echo "powersupersave" > /sys/module/pcie_aspm/parameters/policy
Writing to the above file instructs the kernel to change the ASPM policy to the one with most aggressive power-saving. That's all you need to do in the software. The rest will be taken care by the kernel driver and the pcie controller hardware state machine. That means if both the link partner supports L1 ASPM and there are no pending transactions, the link will automatically go to L1 sub-states.
Please have a look at the L1 Substates Entry and Exit flows below to understand what happens inside the linux kernel after enabling ASPM :-
-- L0 to L1 Sub-states Entry
-- L1 Sub-states to L0 Exit
The Exit from L1ss to L0 can be initiated by either Upstream or Downstream port. We have shown below how downstream port initiates the exit: -
Quick look at the PCIe analyzer traces
In this section, we will have a quick look at the PCIe analyzer traces to give you a real sense of PCIe world. These traces were captured via Teledyne Lecroy PCIe Analyzer. It has proven to be extremely useful for the TLP and DLLP level system troubleshooting/debugging. These traces help us to verify if the link indeed goes to L1 sub-states.
-- LTSSM Flow depicting L0 state at packet 1204
-- LTSSM Flow depicting L1 state at packet 1231
-- L0 state depicting memory TLP transactions on the analyzer
-- L1 State depicted at packet 1231
-- PM Active state request sent from Downstream to upstream
-- CLKREQ Deasserted for L1.1/L1.2 at packet 2822
