Quick-Step veteran Gilbert was 57.2km from the finish and in the lead of the 218km-long stage 16 when he negotiated a bend badly and flew head-first over a wall.
Although he was helped back onto his bike to finish the stage, won by French teammate Julian Alaphilippe, Gilbert emerged from the X-ray truck suffering from his endeavors.
His team said that after further tests at a hospital in Toulouse, the fracture was discovered, meaning he had to pull out of the race.
“I want to say that I’m happy to be here after that tough moment,” he said.
“This isn’t how I wanted to finish my Tour and leaving it like this really hurts.”
Quick-Step bosses panicked when Gilbert went over the wall — it was on the same descent, albeit several kilometers below, that Italian Fabio Casartelli died in a horrible crash in 1995.
Negotiating a left-hand bend at speed, Gilbert failed to brake in time, skidded and was sent flying head first over a parapet to land on rocks several meters below.
Television pictures showed officials helping him to climb back out and onto the road, where he jumped gingerly back on his bike to finish the race.
“We played it well. I attacked to put a little bit of pressure on our group, I wanted to help Julian,” Gilbert added.
“Tactically, we did well, but I crashed and it was all my fault. I simply took the wrong line.
“I fell onto rocks and, when I landed, I thought I’d been broken apart. I’ve had a lucky escape.”
The race ends Sunday on the Champs Élysées in Paris.