Culture

Zwift’s plan for policing pro races continues to shift. Here’s why

Zwift's policy for verifying pro results is getting another facelift after pushback from some riders. Could the platform's Olympic ambitions hinge on it developing a consistent verification program?

Megan Rathwell was breathing so hard that she worried she might throw up.

Rathwell had just won the Yorkshire Grand Prix, the opening round of the 2020 Zwift Classics, and the effort had pushed her to her physical limit. According to data from the race, she had achieved her career-best power output for five- and 20-minute efforts, holding 6.1 watts/kilogram and 5.6 w/kg for the two-time benchmarks.

In the days following Rathwell’s performance, however, her joy turned to frustration and then embarrassment. Her result was overturned by Zwift after she failed the company’s results verification process; officials told her that glitchy equipment likely gave her an unfair advantage.

After losing her result, Rathwell then watched as online commenters left disparaging messages on her personal Strava page.

“Somebody was claiming my boyfriend must have done the race for me. Someone else was looking at my heart rate data and saying I must be cheating — it’s weird when a stranger is commenting on your riding data” Rathwell told VeloNews. “The whole thing has been really embarrassing because I was not trying to cheat at all. I’ve had to deal with a lot of negativity and it’s just been really stressful.”

Rathwell’s situation highlights a nagging problem that Zwift has faced with accuracy and results verification in its races for professional teams and riders. Zwift has gone to great lengths to ensure that the performances in these elite races are, indeed, fair. Yet results in these events have been overturned, and in several cases, Zwift officials pointed the finger at faulty power meters or smart trainers.

The process that Zwift uses to verify these results requires riders to hit similar power numbers in training as they did during the race in question. This process can take several days to complete, and it stands at odds with a quality of virtual racing that both riders and Zwift officials often brag about: that riders can push themselves harder in a virtual scenario than they can in real life.

“I explained to them that I don’t think I can hit the same power numbers in a test on [the road] because it’s not a race,” Rathwell said. “That’s like asking Usain Bolt to go out and do his 100-meter world record in practice to prove it.”

Now, Zwift is writing new rules for pro racing that officials believe will overcome the hurdles present in its post-race verification process. Some of these new rules have been put into place for the Zwift Virtual Tour de France, while others will debut in future professional events. The new rules stem directly from the experiences that Rathwell and other riders had during Zwift’s recent pro events.

Policing pro races

Pro cycling’s collective gaze focused on Zwift’s budding professional scene this spring after the coronavirus pandemic canceled outdoor racing across the globe. Seemingly overnight, WorldTour racers and pro teams flocked to the six-race Zwift Classics series, which paid out $30,000 in prize money and was streamed live on YouTube.

Then, Zwift’s follow-up pro race, the Tour for All, attracted WorldTour teams and was broadcast on Eurosport and GCN.

The boost in global attention represented an enormous leap for the company and its long-term goal of one day becoming an Olympic event.

“Our big aspiration is the 2028 Olympics,” co-founder Eric Min told VeloNews in 2019. “It’s in our backyard in Los Angeles and we are working toward that.”

moolman-pasio-profile-pain-cave
CCC-Liv pro Ashleigh Moolman-Pasio was among several pro road racers to compete in the Zwift pro/am series this spring. Photo: CCC-Liv

An important cornerstone in Zwift’s Olympic ambitions is ensuring that race results at the pro level are fair. Allegations of cheating have swirled within the Zwift community for years, and in 2019 the company weathered a high-profile scandal that led to the disqualification of the British national Zwift champion.

“It’s extremely important that the things people see on the broadcast are real, that it’s something we can trust,” said Stephen Chu, Zwift’s vice president and general counsel. “That’s why we have put all of these resources towards it.”

In years past Zwift races were governed by ZADA (Zwift Anti-Doping Agency), a community-driven group made up of volunteers who scanned Zwift for rides surpassing 5 w/kg. As Zwift honed in on its professional racing ambitions, it added serious bite to the group. In 2019 Zwift staffed ZADA with eight full-time employees and rebranded the group the Zwift Accuracy and Data Analytics.

These days ZADA officials watch the pro/am races and then the pore over power data from the athletes. They look for spikes in power output, moments when a rider’s heart rate does not match her effort, and other data points that appear suspicious

“Our reviewers are purposely anonymous so that people don’t know they work for us,” Chu said. “Some are Ph.D.s and some are analytics guys. The professional experience we have with them is amazing — this isn’t just a bunch of guys who won a Cat 2 race.”

Chu helped Zwift develop pre- and post-race protocols to help speed along ZADA’s policing. Pro Zwift riders weigh themselves on camera before the events, and agree to wear heart rate monitors during the race. After the race, ZADA officials scrutinize power files from the top-three finishers and two randomly selected riders after an event.

If ZADA sees an anomaly, they ask the rider to submit historical power data for one-, five-, and 20-minute power tests. And if they seek a final check, ZADA asks a rider to perform a series of performance tests outdoors on the open road; a final test involves a rider completing a performance test on Zwift’s Three Sisters route.

WorldTour pros flocked to the Zwift Tour for All in May. Photo: Photo: Zwift

Such a system may seem failsafe, but early in the 2020 pro/am season ZADA did overturn several results. Four riders had their results scrubbed in the Yorkshire Grand Prix race alone for various reasons.

According to Chu, the riders in question were not suspected of cheating; rather ZADA officials deemed their trainers or power meters to be out of calibration, or other faulty gear to be the culprit. Riders were not punished by Zwift, and instead, ZADA categorized the decisions with the official wording, “No Intent to Gain Advantage.”

“A lot of times riders don’t know that their equipment isn’t functioning properly, and I’d say nine out of 10 times the result getting changed is because the trainer isn’t calibrated right,” Chu said. “It’s not their fault and we help them get it fixed.”

These decisions represented a gray area for both riders and fans of the sport. Very rarely in Olympic sports are results overturned without officials levying a suspension or punishment against an athlete. Zwift has, in essence, created a new realm of sports governance that takes a no-harm, no-foul mindset.

In fact, Zwift officials do not refer to these instances as “disqualifications,” but rather “annulments.”

“The verbiage we use is that a result was ‘annulled,’ and it’s usually due to hardware,” said Charlie Issendorf, Zwift’s race director for pro/am events. “We never want it to just appear that it is cheating because I don’t think people are intentionally doing this. It’s more of a calibration issue than anything.”

A question over equipment

There are ways to manipulate power meters to boost a rider’s power output. Some power meters allow users to add watts through software; the reading from pedal-mounted power meters can be manipulated by changing one’s crank length.

“There are many ways if you intentionally try,” said Pat Warner, senior vice president at Stages Cycling. “If people change it by a few percent and leave it that way, you get a trend. I don’t know how you set a protocol for that.”

Inside Zwift, the discussion of how to police for devise manipulation has gone on for years, and the launch of professional racing added a new dynamic to that discussion. A simple solution would be to require all pro riders to use the same make and model of smart trainers or power meters for all races, similar to what Zwift does at its live races. The company has shot down that idea, arguing that restricting hardware is against the company’s policy of inclusion.

Favero Assioma app
Many power meters ask users to perform a manual zero offset regularly to ensure accuracy. Photo: Greg Kaplan

One idea argues for all riders to use two power sources to record their performance: a smart trainer and a power meter. Another proposed plan is to build a power passport for the small collection of pro riders, similar to the biological passport used by the World Anti-Doping Agency.

“The passport is ultimately where we want to go,” Chu said. “We know the equipment you’re on, what a normal number can look like, so once you do the race we know if it’s legitimate or not. Everything is done on the front end.”

But that program requires time, as well as a select group of riders to participate. And as the 2020 Zwift Classics and Tour for All series progressed, more and more elite riders flocked to the pro series.

And more riders joining the program meant more smart trainers and power meters for Zwift to police for accuracy and proper calibration. Thus, Zwift opted not to launch a new verification program after the Zwift Classics.

There’s a simmering debate within the cycling community about the accuracy of power meters and smart trainers, and whether power readings produced from different models from different brands are the same. Warner says he has faith that accuracy readings across power meter brands are accurate.

“I think that most brands can make a pedal or spider or crank that in an indoor environment can handle one or two percent [error] if the user uses it as described,” Warner said.

Stages Shimano XT
Pat Warner of Stages Cycling says power meters are accurate, so long as users adhere to manufacturer requests for zero offset calibration.

But he added an important caveat — riders must follow manufacturer guidelines for zero offset calibration in power meters and spin-down calibration in smart trainers. If not, errors in calibration can, over time, occur.

“If you’re trying to be fair, a zero-offset or spin-down should happen after every event,” Warner said. “For us, if you don’t do a zero reset after you install a left power meter, the result is at risk.”

Even minute inaccuracies can have a massive impact in Zwift pro racing, Chu said. The growing strength of the men’s and women’s peloton has meant that riders are closer than ever before in strength. So even a few extra watts here and there can give a rider a clear advantage.

“An extra 10 watts is a lot in a pro race,” Chu said. “We’re not talking about one sprint for 10 seconds, but rather how much energy was saved in a one-hour race if you’re getting that additional power.”

An extra boost of power is what Chu and his team determined had helped Rathwell achieve her stellar result at the Yorkshire Grand Prix.

Disqualified

Rathwell retired from her professional road career in 2016, yet she continues to ride and race as she pursues a graduate degree in Civil Engineering. Rathwell began riding on Zwift in 2019, and the Yorkshire Grand Prix marked her first foray into pro competition.

She said she followed the proper pre-race protocols and wore a heart rate monitor and submitted a video of her standing on a scale to verify her weight.

“I was using my new trainer which I’ve only had for a few months, and I started the race confident that everything I was doing was correct,” Rathwell said. “In the race, I just went super hard.”

Indeed, Rathwell rode at or near the front of the women’s event for the majority of the event, which was held as a points race format. While other riders surged past her to gobble the points, Rathwell pedaled ahead, eventually breaking away on the course’s long climb. She crossed the finish line alone, 30 seconds ahead of the competition, before slumping down onto her bicycle in exhaustion.

The next day Rathwell received the news that Zwift was looking into her result. Later that day, a Zwift official said in an email that her performance had “pushed the limit of human physiology.”  The news was a psychological blow; she had trusted the wattage reading on her Zwift screen and had been overjoyed by her performance output.

“If I had done 400 watts for 40 minutes then obviously it’s wrong,” Rathwell said. “I just didn’t think the numbers I was putting out felt crazy so I was really disappointed.”

Zwift’s verification process called for Rathwell to perform a performance test on outdoor roads. Rathwell set out on the rolling terrain around Victoria, British Columbia, but the lack of a long and sustained climb presented a challenge for the longer 10-minute effort.

“You don’t realize how hard it is to put down consistent power on an undulating road,” Rathwell said. “I did what they asked and they said I hit the [one- and five-minute tests] but that the data for my 10-minute wasn’t high enough.”

So, a few days later Rathwell completed the ZADA power verification test on Zwift, which consisted of one-, five-, and 20-minute FTP tests. Rathwell said Zwift officials told her that the performance was strong, but not good enough.

Rathwell’s result was scratched and the stage win went to Cecilia Hansen of Team Heino; the change in results was sent out via a press release a week after the event. Rathwell was disappointed but not deterred. Zwift officials told her that her new Wahoo Kickr Core had likely added a few watts to her effort, and they asked her to use her Stages power meter for the next race.

During the next race, Rathwell said the Stages power meter kept cutting in and out over Bluetooth, which caused her to be dropped from the group during the event. Zwift officials told her that wifi interference with her other devices was likely the culprit, she said.

“That kind of ruined it for me,” Rathwell said. “Since then it feels like all I’ve been doing is trying to troubleshoot.”

Stephen Chu, Zwift’s vice president and general counsel.

Rathwell has continued to participate in Zwift races, yet she said the company’s post-race protocols for verifying her result left her feeling uneasy. In her opinion, asking a rider to replicate a performance in training from a racing scenario ignores a basic understanding of the sport.

I’m very motivated by racing situations,” Rathwell said. “I push myself harder to stay with others than I would do just riding by myself.”

Rathwell is not alone. VeloNews spoke to another rider whose results were annulled during a 2020 Zwift pro/am race after failing the post-race verification protocols. This rider, who asked not to be identified, said Zwift officials also blamed the smart trainer for the annulment.

The process to verify the result, however, raised more questions than answers.

“You seemingly can’t target a Zwift race to produce your best performance because unless it’s at a live event you have to be able to prove it outside as well, or you’ll be [disqualified],” the rider said. “Like it or not inside and outside riding are different.”

That’s a sentiment that’s been repeated multiple times in the past year by the growing number of elite riders who have raced on Zwift. In a virtual simulation, riders must only focus on their power output and not the road or the other riders around them. With the absence of wind, potholes, or the jostling of elbows in the pack, riders can simply push themselves harder than they can on the road.

And the disconnect between this reality and Zwift’s testing protocol has had an impact on the riders. Rathwell continues to race but says her passion for pushing herself on Zwift has evaporated.

The rider who spoke to VeloNews said the ZADA process simply sapped their motivation to race.

“The whole ZADA process left me empty and out of love with Zwift which is a huge shame,” the rider said. “Whenever I’ve attempted to ride Zwift since I’ve felt angry and constantly questioned if the numbers I was seeing on screen.”

Pre-race verification

Chu read emails from Rathwell and other riders throughout the verification process. In each message, Chu said, he could tell that the riders were upset with the result annulment, even when Zwift officials said the culprit was equipment and not cheating.

Some riders were upset with ZADA’s way of communicating the bad news. Others echoed Rathwell’s sentiment that the annulments were embarrassing, and raised unjust scrutiny from the cycling community.

While Chu still trusted ZADA’S post-race verification procedures, he realized after the Zwift Classics that Zwift needed a new method.

“Even if we sugarcoated it the best way we could I think there’s a level of disappointment that we can’t overcome,” Chu said. “It’s an experience that is jarring and has definitely impacted their relationship with us, which hasn’t surprised me at all.”

In lieu of building a passport for every rider, Chu and his team came up with another plan. Why not have riders complete a performance test before the races, and not afterward? Such a test could create baseline data for each rider, and also test a rider’s gear for glitches.

The short window between the Zwift Classics and the Tour for All prevented ZADA from building such a test for the WorldTour race. But in the weeks afterward, ZADA developed a new standard that has yet to be employed.

Zwift is skipping post-race verification during the Virtual Tour de France.

Riders will again video record a weigh-in, and now they will also video record his or her height. Then, prior to their first pro race, every rider in the field must complete a 1.5-hour performance test on Zwift that is recorded by both a smart trainer and a secondary power meter.

“The test is more about analyzing the equipment than analyzing your best efforts,” Chu said. “And if we can have the entire peloton verified, that would be cool.”

Going forward, all Zwift pro racers must dual-record with that same power meter and smart trainer in the races.

Whether or not the new method works is a question for the future. A Zwift representative said the company is using a partial version of this new verification process for the Virtual Tour de France, but due to the event’s focus on charity, the company is using a “lighter touch” on the pro riders, and not making them undergo post-race verification tests.

How Zwift’s new protocols hold up under normal professional competition is yet to be seen. Chu is hopeful that the procedures work, however he acknowledged that the verification process may change again.

“Why do we keep changing the rules? We are learning as we go along,” Chu said. “Every single series is a laboratory and it’s a chance for us to get better.”

The future of this protocol could have vast ramifications on Zwift’s Olympic ambitions. After all, Olympic sports are built on trust, specifically the trust that fans have in the fairness of the competition. There’s another level of trust that is equally important, however. Athletes must trust that the system governing them is also fair.