The difference between mathematics and physics is in the interpretation. The equations of physics and their solutions are just math—it's the interpretation in terms of everyday experiences that makes them physical. Yet in practice the line between math and physics is fuzzy. We are not always sure where math stops and interpretation begins. In this respect quantum theory is definitely the fuzziest—there are several different interpretations of its equations. I'll come to this topic later.
The distinction between mathematical devices and "reality" became a real problem when Copernicus published his heliocentric theory in 1543. He did it in a book titled "De Revolutionibus Orbium Coelestium," which means "On Revolutions of Celestial Spheres." Celestial spheres were introduced by ancient Greeks—they were the mainstay of the geocentric system. Copernicus was probably afraid to outright dismiss the theory that was officially accepted by the Church, of which Copernicus was a functionary. In his book he described his heliocentric calculations merely as a mathematical device to improve the calculation of epicycles.
Copernicus was right to be scared. Not long afterwards, his follower, Galileo, was punished by the Church for spreading the news of the heliocentric theory. According to the apocryphal story, after his trial Galileo said, sotto voce, the famous words eppur si muove (and yet it moves). If we believe the story to be true, it would be the first record of somebody taking Copernicus's "mathematical device" at face value, as a reflection of physical "reality." Galileo in effect expressed his belief that, since the heliocentric calculations are simpler than the traditional epicycles, they must be closer to reality.
What was the difference between the geocentric and heliocentric theories? The geocentric theory was first formulated by a Greek astronomer, Ptolemy (87–150 AD). Stars and planets were supposed to be attached to a series of spheres rotating around the Earth. This picture worked fine for stars, they indeed seem to travel in concentric circles around the firmament. But planets are strange. Their tracks resemble cycloids rather than circles. Some of them occasionally move backwards with respect to firm stars.
The only possible explanation within the geocentric theory was that there must be small circles called epicycles on the surfaces of the planetary spheres and that the planets revolved slowly around those circles while at the same time the spheres revolved around the Earth (Fig 1.).
How viable is this theory? Looking at the predictions it made of planetary movements, it is pretty good. One could probably derive it nowadays from the heliocentric theory by changing the system of coordinates (since the system attached to the Earth is not inertial, one would have to use Einstein's general relativity to do that correctly). Maybe physicists would be forced to introduce more cycles upon cycles to account for all the anomalies—maybe infinitely many. So even though the two theories differ in complexity, they are presumably equivalent in their predictive power.
Why don't we stick to the geocentric theory? For one, the simplicity of the heliocentric theory made it possible for Newton to use the idea of gravity to explain the movements of planets. The forces acting on spheres and epicycles are much more complicated and couldn't be easily explained by gravity (in the geocentric coordinate system, one would have to consider the centrifugal force and the Coriolis force in addition to gravity).