Choosing a test system for emerging PCIe 6.0 designs

New PCIe 6.0 standards demand instruments with upwards of 59 GHz bandwidth and which are capable of running special signal integrity tests.

Contributed By Anritsu Co.
PCI Express (PCIe) 6.0 addresses the high-speed data transmission needs of emerging applications ranging from data centers to connected cars. With the doubling of data rates and upgrades of performance specifications, PCIe 6.0 adds complexity to high-speed interconnect designs. Engineers need signal integrity tests and analysis tools able to verify products comply with the new PCIe 6.0 standards.

Data rates for each PCIe generation. Click image to enlarge.

When jumping to 64 GT/sec for PCIe 6.0, the PSI-SIG standards committee only slightly altered the compliance requirements. Tighter channel and connector loss and reflection parameters have been implemented to address signal degradation. There were slight improvements in receiver and transmitter equalization as well. There are no major innovations to address the expected complications associated with steeper rise-fall times, narrower unit intervals (UIs), and greater insertion loss associated with doubling the data rates.

PCIe 6.0 utilizes 32 Gbaud PAM4 signaling. Though the underlying frequency is the same as the PCIe 5.0 specification, there is extra circuitry and logic involved for the PAM4 mode to track three eyes, along with the logic changes needed to operate in Flow Control Unit (FLIT) mode. FLIT was selected because it allows error correction on fixed-sized packets. Because error correction happens on FLIT, the cyclic redundancy check (CRC) and retry must take place at the FLIT level.

In addition to PAM4, the PCIe 6.0 specification includes error assumptions, including correlation between errors on a lane, as well as across lanes. PCIe 6.0 uses a special approach to maintain low latency through a combination of relatively lower First Bit Error Rate (FBER) combined with a light-weight, low latency Forward Error Correction (FEC) for initial correction. A robust CRC then detects any errors that remain after correction. The result is a link-level retry, which is also low latency.

Unlike networking standards that have 100+ nsec of FEC latency, PCIe technology is a load-store protocol, dividing actions into those associated with memory and registers. Therefore, it must strictly maintain all specifications, especially when it comes to latency, power, and high bandwidth. Using FEC and CRC allows PCIe 6.0 to realize the specified low latency with latency reduction in most cases. It also realizes low complexity and a low bandwidth overhead.

PAM4 effect on BER

PAM4 signaling alleviates channel loss because it runs at half the frequency with two bits per UI. Because PCIe 6.0 has three eyes in the same UI, however, eye height and width are reduced. As a result, the BER will be several levels of magnitude higher with PAM4, which is why FEC is necessary.

For PCIe 6.0, BER is a combination of the FBER, correlation of errors in a lane, and correlation of errors across lanes. There are two primary mechanisms to correct the errors in a lane and those across lanes. The most notable are through FEC and detection of errors by CRC, resulting in the eventual correction through link-layer retry. FEC operates on the principle of sending redundant data that can be deployed to correct some errors at the receiver. CRC is an error detection code used to authenticate packet transmission between the sender and the receiving end.

PCI-SIG established a low-latency FEC of below 2 nsec for PCIe 6.0, and that is to be part of the specified overall signal latency of below 10 nsec. FEC is based on a fixed number of symbols, making it simple to transition to FLITs, as they are fixed size as well. Link frequency is 64 GT/sec.

FEC logic can be run at any frequency. The expectation is that the logic will operate at 1G (or 500 MHz or 2G) and easily reach a latency exceeding 2 nsec. PCI-SIG recommends a lightweight FEC for correction. The robust CRC for detection, combined with a fast link-level replay, handles any errors that the FEC cannot correct. As long as the replay probability of a FLIT is approximately 10-6, there is no appreciable performance impact either from the FEC latency or the replay latency in case of an undetected error. A combination of FEC correction and CRC detection results in a replay that effectively corrects nearly all common errors.

A recommended approach is to establish a FEC symbol error threshold. By doing so, engineers have broader control over error conditions that affect patterns during capture by ignoring insignificant events that are normally corrected in the FEC environment.

To set a threshold, a bit error rate tester (BERT) generates a PAM4 signal to the Device Under Test (DUT) receiver input. The DUT determines the logic state of the input signal and loop decision to transmitter output for the error signal in the BERT for analysis. The BERT’s built-in Error Detector (ED) determines if the DUT’s decision was correct. For relevant results, the BERT’s jitter and noise profiles must comply with standards.

Link equalization

Link training and stressed receiver tolerance are simultaneously evaluated using a stressed signal in the link equalization test. Two tests–for receivers and transmitters–must take place using SigTest software developed by the PCI-SIG.

Though similar to a standard stressed-eye receiver BER, receiver link equalization has one notable difference. The DUT must first perform link negotiation to correctly compensate for the test channel. The idea of stressed receiver tolerance testing is to send the DUT-receiver the worst-case signal that still complies with the specification.

The BERT PPG differential output is transmitted to both the DUT-receiver and the oscilloscope. The DUT-transmitter output goes both to the oscilloscope and the BERT ED, which acts as a reference receiver. Click image to enlarge.

Prior to conducting the test, the signal transmitted from the BERT must be precisely calibrated to mimic the worst-case signal at the end of the test channel. The test signal has jitter and interference impairments that include random jitter (RJ), sinusoidal jitter (SJ), sinusoidal differential mode interference (DMI), and common mode interference (CMI),

The BERT PPG differential output is split so the signal goes to both the DUT-receiver and the oscilloscope. The DUT-transmitter output is also divided so its signal goes to the oscilloscope and the BERT ED, which acts as a reference receiver.

The BERT PPG sends requests to the DUT-transmitter through the PCIe physical layer logic-sub-block protocol. Click image to enlarge.

Transmitter link equalization is a required compliance test that verifies the device correctly changes equalization within the specified time when the link partner requests it. The BERT requests an equalization change from the devices that simultaneously sends a trigger to the oscilloscope so the time delay to the DUT can be measured in the electrical domain.

The BERT PPG sends requests to the DUT-transmitter through the PCIe physical layer logic-sub-block protocol. The BERT PPG sequentially sends requests to the DUT-AIC for every FFE preset at each PCIe data rate. The DUT-transmitter modifies its FFE scheme and transmits the signals. The DUT-transmitter output is split so its signal goes to the oscilloscope and to the BERT ED. The oscilloscope observes the high-level equalization change while the ED serves as a reference receiver that confirms the preset change.

The BERT uses the PPG auxiliary output to trigger the oscilloscope acquisition of each signal. The oscilloscope captures the waveforms with every FFE preset and every data rate. It then runs SigTest to evaluate each waveform according to the compliance requirements. Reports on the results can also be created through SigTest.

Selecting the proper test system

To conduct these measurements accurately, PCIe 6.0 test systems need a feature-rich, protocol-aware BERT and an oscilloscope. The BERT needs a built-in instrument-quality PPG that can apply precise levels of specific signal impairments and a built-in ED capable of verifying compliance with the PCIe specifications. The BERT should have multiple NRZ pattern-generating channels and error detectors that operate at 32 GT/sec and PAM4 channels at 64 GT/sec to support PCIe 6.0 and earlier generations.

Low intrinsic jitter of 115 fsec and 12 psec 20 to 80% rise/fall times are also necessary for signal integrity. The BERT must apply every required signal impairment in amplitude ranges that exceed those required by the PCIe 6.0 specifications,

Certain BERTs have FEC analysis functions for the built-in ED. The functions leverage high input-sensitivity performance or the ED to detect FEC symbol errors based on the 400 GbE FEC standard. Bit error changes and FEC symbol errors with alterations in input amplitude and jitter conditions can be monitored in real-time to quickly and reproducibly conduct evaluations when symbol error counts exceed the correction ability of FEC.

The oscilloscope should have real-time sampling bandwidth exceeding 50 GHz. For both transmitter signal evaluation and calibration of stressed-eye receiver tolerance tests, the oscilloscope must also support PCI-SIG test software analysis tools.

Test systems must have this performance level to verify PCIe 6.0 designs in emerging applications. Two notable applications are data centers and automotive.

Data centers now must support next-generation, high-speed, large-capacity 5G mobile communications. To do so, they are installing equipment meeting the 400 GbE communications standard. There is also investigation into 800 GbE and 1.6 TbE standards to facilitate faster speeds. PCIe 6.0 will be the connectivity technology used for these emerging high-speed designs.

Automotive applications essentially require data center-class computing power and home-theater complexity. PCIe 6.0 technology is enabling the automotive infotainment and connectivity ecosystem, including critical safety applications such as Advanced Driver Assistance Systems (ADAS). It is becoming the connectivity of choice because it features fault tolerance by multiple error correction system architecture, secured interoperability, and high bandwidth.

All in all, PCIe 6.0 is creating new design challenges that require sound testing processes. The basic test system required is comprised of a protocol-aware BERT with FEC functionality and a high-speed oscilloscope that produces high-quality eye diagrams and has comprehensive analysis tools.