As 5G and IoT interconnected devices and the associated high bandwidth requirements are expected to rise significantly, data center operators will need to migrate to higher bandwidth networks that exceed the 100GB Ethernet (100GE) commonly used today. Migrating to next-generation 400GE networks requires faster memory and higher-speed serial bus communications. In addition to upgrading the Ethernet interface to 400GE, the server also needs to use a higher-speed serial expansion bus interface and memory.

The PCIe (PCI Express) expansion bus is now migrating to the latest standardized PCIe 5.0, also known as PCIe Gen 5. At the same time, DDR (Double Data Rate) memory is also migrating from DDR 4.0 to DDR ≈ 5.0. The PCIe Gen 5 specification is a fast-moving enhancement to the PCIe 4.0 standard developed by the PCI-SIG. PCI-SIG is a standards body that defines all PCIe specifications. With the finalization of the PCIe 5.0 Plug-In Electromechanical (CEM) specification, the PCIe 5.0 standard was recently completed and released in June 2021, a companion piece to the existing PCIe 5.0 base (silicon) specification released in 2019.

The evolution of the PCIe standard doubles the transmission speed

The original parallel PCI bus was introduced in 1992 to expand the capabilities of personal computers, allowing the addition of graphics and network cards and many other peripherals. PCIe is a high-speed serial bus designed to replace PCI and other existing legacy interfaces such as PCI-X (PCI eXtended) and AGP (Accelerated Graphics Port). PCIe not only has high throughput, but also is small in size, and the link width can be expanded between ×1, ×2, ×4, ×8, and ×16. PCIe is based on a point-to-point bus topology between the root complex (system/host) and endpoints (plug-ins), supporting full-duplex packet-based communication.

　　

PCIe Duplex Link Communication

The PCIe 1.0 standard came out in 2003 and provided a rate of 2.5G transfers/second (2.5GT/s). PCIe currently provides a rate of 2.5GT/s~32GT/s. PCIe 5.0 doubled PCIe 4.0 transfer rates from 16GT/s to 32GT/s, but didn’t offer any new additions, as the goal at the time was to provide additional speed in the shortest amount of time.

All PCIe standards released today employ non-return-to-zero (NRZ) signaling. However, the PCI-SIG is currently developing the PCIe Gen 6 specification, which will again double the transfer rate to 64GT/s, migrating away from NRZ signaling. The Gen 6 sixth-generation specification will use PAM-4 signaling and low-latency FEC (forward error correction) technology to improve data integrity.

All PCIe standards must be backward compatible, that is, PCIe 5.0 (32GT/s maximum data rate) must also support 2.5GT/s, 5GT/s, 8GT/s, 16GT/s, and 32GT/s.

PCIe Specification Timeline

PCIe lanes and link speeds

PCIe compliance testing with specific challenges

PCI-SIG is the developer of non-proprietary PCI technology standards and related specifications, and PCIe is now the de facto standard for servers. The PCI-SIG defines the PCI specification to support the required I/O functions while being backward compatible with previous specifications. To enable adoption of PCI technology across the industry, the PCI-SIG supports both interoperability and compliance testing, including the tests that must be performed and passed to achieve compliance.

PCI-SIG allows members to conduct interoperability testing against other member products and test suites, and participating products either pass or fail the test. To pass formal conformance testing, products must pass at least 80% of interoperability tests and pass all standard conformance tests.

PCIe 5.0 faces specific challenges. With a maximum data rate of 16GT/s, PCIe 4.0 is a speed-enhancing specification for the previous generation of PCIe and has proven to be more difficult to implement than previous standards. In PCIe 5.0, both computer PCIe lanes and motherboards face significant challenges as they deal with 32GT/s data rates. In addition to the challenges encountered at lower data rates, PCIe 5.0 devices are expected to experience significant signal integrity challenges. Tektronix has PCI-SIG approved test suites for all data rates (Tx, Rx and PLL bandwidth).

　　

Tektronix PCIe Gen 5 Tx Compliance Test Solutions

Tektronix is ​​a major contributor to the PCI-SIG, has made significant contributions to the PCIe 4.0 and 5.0 physical layer test specifications, and has done extensive road-seeking experiments to define PCIe 6.0 Tx/Rx measurement methods. Tektronix also played a key role in conformance and interoperability testing during PCIe standard development and implementation.

For PCIe 5.0 transmitter testing, proper test equipment and automation software are essential

When developing a PCIe Gen 5 transmitter device, either at the base (chip) level or at the CEM (system and plug-in) level, chip-level verification (usually performed by PHYIP companies) and pre-compliance testing will be required before the Devices are submitted to PCI-SIG for formal compliance testing. Therefore, obtaining appropriate test equipment and associated automation software is critical.

PCIe compliance testing includes:

Electrical Test – Evaluation Platform, Plug-In Transmitter (Tx) and Receiver (Rx) Features

Configuration Testing – Evaluating Configuration Space in PCIe Devices

Link Protocol Test – Evaluate the device’s link-level protocol characteristics

Transaction Protocol Test – Evaluate the device’s transaction-level protocol features

Platform BIOS Test – Evaluates the ability of the BIOS to identify and configure PCIe devices

In terms of electrical testing, it is divided into two sets of measurements, one at the basic level and one at the CEM level. These tests are further divided into standard tests and reference tests:

　　

PCIe Basic and CEM Compliance Measurements

　　

Eye

Both types of measurements require a high-bandwidth real-time oscilloscope capable of capturing data waveforms. Post-processing techniques are then employed to make the corresponding voltage and timing measurements required in the base specification and the CEM specification. Uncorrelated jitter examines the jitter inherent in the system after removing packet and channel intersymbol interference (ISI). In addition to jitter, the oscilloscope makes eye height and eye width measurements. A number of “conformance test patterns” are specified in the base specification. A waveform record containing multiple occurrences of the entire compliance test pattern is recommended to construct a representative eye diagram.

In basic Tx testing of the device, the specification states that measurements are made directly at the pins of the transmitter. If direct access is not possible, test points should be as close as possible to the device pins. If the user has a good understanding of the S-parameters, any splice channel loss can be de-embedded by physically reproducing the channel or emulation. Starting with the 4.0 specification, another de-embedding technique is described, applying CTLE (Continuous Time Linear Equalization) to uncorrelated jitter measurements during waveform post-processing, which effectively eliminates ISI up to the pin.

　　

Tx equalizer presets

Any PCIe 5.0 product submitted for PCI-SIG certification must successfully pass compliance testing using the specified Tx equalizer settings presets, supporting speeds from 2.5GT/s up to 32GT/s. These presets are used to equalize intersymbol interference caused by frequency-dependent attenuation differences within the code stream, improving signal integrity. Each preset is a specific combination of undershoot (before the cursor) and de-emphasis (after the cursor).

There are various specific implementations that allow the DUT transmitter to scan through various data rates and TxEQ presets. However, the base specification specifies a common method in which a 100MHz clock burst is delivered to lane 0 of the receiver. This can be done automatically using an arbitrary function generator (AFG).

For PCIe links with a maximum rate of 32GT/s, there are new verification challenges for base clocks (Refclks). The base specification has expanded the jitter limit in proportion to the data rate, but Gen 5 has lowered the limit disproportionately to 150fs. This high-frequency jitter measurement requires proper application of the common clock transfer function and consideration of worst-case transfer delays. This latest version of the specification also pushes measurements from a basic level specification (chip level) to a CEM specification requirement (appearance level), which must meet compliance testing.

　　

CEM plug-in PCIe 5.0 conformance test and automatic preset switching

Tektronix PCIe solutions for more confidence in compliance testing

Oscilloscope bandwidth and sample rate requirements. For the basic Tx test, each PCIe 5.0 lane runs at 16GHz (because two bits can be sent in one cycle), and the third harmonic reaches 48GHz. Since there is not much valid signal information above the third harmonic, PCIe 5.0 basic Tx testing only requires a real-time oscilloscope with 50GHz bandwidth. For CEM Tx testing, measurements are made near the end of the worst-case channel, reducing high frequency content and requiring a 33GHz bandwidth. To ensure adequate waveform post-processing (SigTest), a minimum of 4 points per unit interval is required, and CEM allows up to 2xsinx/x interpolation, so the minimum sampling rate is 128GS/s.

Automatic conformance testing. In conformance testing, performing analysis manually is time-consuming and error-prone. To save time, it is best to use automated software, which not only reduces the workload, but also speeds up compliance testing. For electrical verification, PCI-SIG offers SigTest offline analysis software, which performs analysis using data acquired by an oscilloscope. The automation software also controls the device under test (DUT), using an arbitrary function generator as the pattern source, allowing the DUT to automatically pass through the various speeds, de-emphasis, and presets required for compliance testing.

A complete round of compliance testing requires the acquisition of multiple waveforms per channel at different DUT settings. This set of waveforms will be increased by the number of channels (up to 16) that need to be analyzed. The ability of software to manage and store data for analysis and future reference requirements is an important metric for any compliance test solution. The automation software can also adjust the oscilloscope horizontal and vertical settings and acquisition. In addition to configuration and analysis, multiple acquired waveforms can be managed using automation software.

The automation software can select the data rate, voltage swing, presets, and tests to perform. It can also provide options to embed package parametric models, de-embed cables, test fixtures, or other elements required to reach the target test points specified by the specification. Analysis results from the software can often be compiled into a report in PDF or HTML format, which can include pass/fail test summaries, eye diagrams, setup configurations, and user notes.

Using the Tektronix DPO70000SX Series oscilloscope and AFG31252 arbitrary function generator, the PCIExpressGen1/2/3/4/5 solution automates transmitter verification and compliance testing at the basic (chip) and CEM (system and plug-in) levels.

TekExpressPCIe5.0Tx automatic software function:

Auto-step the DUT through different tempo, pattern and Tx EQ presets

Verify that the signal is correct at the transmitter before taking measurements

Perform channel and packet embedding and de-embedding

Support SigTest and SigTest Phoenix software and template files

100MHz Reference Clock Jitter and Signal Integrity Measurements Using Silicon Labs. “PCIe Clock Jitter Tool” and Tektronix DPOJET Software

Historically, when a new generation of PCIe devices entered compliance testing, a significant portion of the devices would fail the first interoperability workshop when they were tested for PHY and link training compliance. Before the PCI-SIG workshop, it is critical to ensure that a complete oscilloscope, AFG, BERT (for Rx testing) and automation software solutions are in place. Tektronix PCIe Test and Debug Tx, Refclk and Rx solutions guide you through compliance testing and debugging prior to interoperability testing, ensuring your designs meet PCI-SIG PCIe standards with confidence.