Losing the Grandmaster at Line Rate: Testing TSN Failover Where It Actually Breaks

A gate schedule that holds on an idle bench can fail the moment the time reference disappears while the egress queues are full. Idle traffic hides clock drift, because when a window slides there is nothing on the wire to compete with. Under load, a window that opens late means a control frame arrives behind a best-effort frame that is already being transmitted. A qualification test has to create both conditions at once, or it tells you nothing about failover.

Why idle failover tests pass

Time-aware shaping, as defined in the IEEE 802.1 time-sensitive networking standards, assumes every bridge agrees on time within a tight bound. gPTP provides that agreement and the gate-control list depends on it. When the grandmaster dies, each bridge goes into holdover on its local oscillator and keeps running its schedule against a clock that slowly moves away from its neighbours' clocks. On an empty network a window that is slightly off still passes the scheduled frame, because the port is idle anyway. Fill the lower-priority queues and things change: the guard band ahead of the protected window was sized for a known misalignment, and once the accumulated drift goes past it, an interfering frame is still on the wire when the gate opens, the scheduled frame waits, and if the window closes before the frame fits, its latency grows by a full cycle. That is the failure mode to hunt. No loss counter moves, no link drops, yet a brake or steering stream misses its deadline without any alarm.

The test that exposes it

Build the topology with the real number of bridges on the longest path, because drift adds up hop by hop and a two-switch bench will understate it. Load every best-effort and lower-priority class to line rate with maximum-length frames, since the largest interfering frame is what the guard band was sized for. Timestamp the scheduled stream at ingress and egress against a capture reference disciplined independently of the network under test, then remove the grandmaster by cutting its power rather than shutting it down administratively, so that announce timeouts decide the recovery and not a graceful handover. Record the latency of every frame relative to the cycle boundary through holdover, best master reelection and servo settling, and repeat the cut at different phases of the cycle and at the cold and hot ends of the thermal envelope, where oscillator drift is worst. The pass criterion is not "no frames lost". It is every scheduled frame inside its window from the instant of the cut until well after the new grandmaster has settled.

The number that sets the budget

The parameter to fix before writing any procedure is the maximum time error the schedule tolerates between adjacent bridges, around 1 µs in a typical zonal backbone design, and the rest of the test follows from how long the system can stay under that bound without a valid grandmaster.

Choosing between reelection and hot standby

The decision rule is simple: divide that time error budget by the worst-case oscillator drift rate across temperature, and the result is your holdover window. If the measured time from the cut to resynchronisation under full load is longer than that window, best master reelection alone is not enough, and the design needs a hot standby grandmaster with redundant sync paths so bridges can switch time sources without passing through holdover. Reelection takes longer under load than on the bench, because announce and sync messages queue behind traffic unless their priority mapping is right. As an estimate, reelection plus servo settling on default timers takes several seconds, far longer than a low-cost oscillator can hold a sub-microsecond budget. Hot standby closes that gap but adds a new failure surface, so the final runs kill the standby as well, then force a split where both grandmasters claim time, and check that no bridge steps its clock in the middle of a window. What remains is a cost trade-off: a better oscillator in every bridge widens holdover and may remove the need for the standby, and only this load test will show which of the two the program can drop.

Back to SDVProtocol · Domains

← Back to SDVProtocol
Engineering & Architecture Brief

Losing the Grandmaster at Line Rate: Testing TSN Failover Where It Actually Breaks

A gate schedule that holds on an idle bench can fail the moment the time reference disappears while the egress queues are full. Idle traffic hides clock drift, because when a window slides there is nothing on the wire to compete with. Under load, a window that opens late means a control frame arrives behind a best-effort frame that is already being transmitted. A qualification test has to create both conditions at once, or it tells you nothing about failover.

Why idle failover tests pass

Time-aware shaping, as defined in the IEEE 802.1 time-sensitive networking standards, assumes every bridge agrees on time within a tight bound. gPTP provides that agreement and the gate-control list depends on it. When the grandmaster dies, each bridge goes into holdover on its local oscillator and keeps running its schedule against a clock that slowly moves away from its neighbours' clocks. On an empty network a window that is slightly off still passes the scheduled frame, because the port is idle anyway. Fill the lower-priority queues and things change: the guard band ahead of the protected window was sized for a known misalignment, and once the accumulated drift goes past it, an interfering frame is still on the wire when the gate opens, the scheduled frame waits, and if the window closes before the frame fits, its latency grows by a full cycle. That is the failure mode to hunt. No loss counter moves, no link drops, yet a brake or steering stream misses its deadline without any alarm.

The test that exposes it

Build the topology with the real number of bridges on the longest path, because drift adds up hop by hop and a two-switch bench will understate it. Load every best-effort and lower-priority class to line rate with maximum-length frames, since the largest interfering frame is what the guard band was sized for. Timestamp the scheduled stream at ingress and egress against a capture reference disciplined independently of the network under test, then remove the grandmaster by cutting its power rather than shutting it down administratively, so that announce timeouts decide the recovery and not a graceful handover. Record the latency of every frame relative to the cycle boundary through holdover, best master reelection and servo settling, and repeat the cut at different phases of the cycle and at the cold and hot ends of the thermal envelope, where oscillator drift is worst. The pass criterion is not "no frames lost". It is every scheduled frame inside its window from the instant of the cut until well after the new grandmaster has settled.

The number that sets the budget

The parameter to fix before writing any procedure is the maximum time error the schedule tolerates between adjacent bridges, around 1 µs in a typical zonal backbone design, and the rest of the test follows from how long the system can stay under that bound without a valid grandmaster.

Choosing between reelection and hot standby

The decision rule is simple: divide that time error budget by the worst-case oscillator drift rate across temperature, and the result is your holdover window. If the measured time from the cut to resynchronisation under full load is longer than that window, best master reelection alone is not enough, and the design needs a hot standby grandmaster with redundant sync paths so bridges can switch time sources without passing through holdover. Reelection takes longer under load than on the bench, because announce and sync messages queue behind traffic unless their priority mapping is right. As an estimate, reelection plus servo settling on default timers takes several seconds, far longer than a low-cost oscillator can hold a sub-microsecond budget. Hot standby closes that gap but adds a new failure surface, so the final runs kill the standby as well, then force a split where both grandmasters claim time, and check that no bridge steps its clock in the middle of a window. What remains is a cost trade-off: a better oscillator in every bridge widens holdover and may remove the need for the standby, and only this load test will show which of the two the program can drop.

Related Architecture Specifications (AUTOMOTIVESDV): careamr.com · chargingorchestration.com · isaccare.com

More Technical Articles on sdvprotocol.com

Configuring IEEE 802.1Qbv Time-Aware Shaper Gate Schedules in Automotive Switches →Migrating from CAN-FD to CAN XL: Physical Transceivers, SIC, and Gateway Offload →Sub-Nanosecond Clock Synchronization: IEEE 802.1AS-2020 gPTP in SDVs →