Cavendish had gone to hospital after world champion Peter Sagan elbowed him into the railings by the side of the road at the finish in Vittel.
Sagan was kicked off the Tour for his part in the crash but his Bora team stated they had appealed against the decision by the Tour's race commission.
Late on Tuesday evening, Cavendish's Dimension Data team posted a picture of the British rider on Twitter with his arm in a sling, revealing that he was pulling out of the Tour.
"I'm obviously massively disappointed to get this news about the fracture," said Cavendish.
