Part of the issue is that most people think that mathematics is a) a single unified whole, b) about numbers somehow, though sometimes using a placeholder such as
x when you don't know what the number is, and c) is some direct line to absolute truth, truth in the sense that is familiar to everyday people.
None of those things are at all the case.
Mathematics is a number of different things, really, but mostly it is a) a language (well, a family of several languages, but bear with me), and b) a protocol about how to use that language politely.
As a language - and when I say this, I don't just mean conventional mathematical notation (which I've already discussed the ideosyncracies of
here,
here, and
here - oh, and throw in
this while you're at it, too) but the whole gamut of terminology and agreed-upon notions - it exists primarily to express concepts. It is a language with a very tightly defined scope, and one which is remarkably compact and dense at times, but it is more than anything a language. As such, it has its idioms and oddities, but within the constraints of its domain of discussion, it can express nearly any idea - regardless of whether it is a reflection of the physical world or not. You can, indeed, say utter nonsense with mathematics, and people do so all the time, but conversely you can say things which sound like nonsense but still have a deeper meaning to them.
And like most other languages, it can easily convey some ideas which are much harder to express in other languages. Indeed this is precisely what the language evolved/was developed to do. At the same time, it struggles to convey certain other ideas which are facile in most other languages, mainly due to the assumptions the language was developed under.
Now, because the language originated as a way of reasoning about certain things (primarily geometry, historically speaking, but by the time of Al Khwarizmi it no longer really had any direct referents to the physical world at all; even in the days of Pythagoras and Euclid, it included such physically absurd ideas as points of one dimension and lines which extend to infinity - absurdities which are, nonetheless, quite useful absurdities) it has a set of rules of order meant to 'keep the conversations on topic' (as it were). However, those rules were not really more than guidelines until Hilbert and Co. started taking them far more literally than was actually sensible to do.
As it happens, it is really useful to use that language - and those rules of
politesse - to
describe things in the real world in a succinct and lucid manner, by making analogies between what you are saying in the mathematical world and what you are observing in the real world.
The problem arises, as is so often the case, when you take the metaphors literally, and ascribe them some sort of deeper meaning, rather than seeing them for what they are - a way of explaining something indirectly. Keep in mind that all abstractions leak, so none of these are ever going to be 'really real'.
Also, you can often have several different metaphors for the same thing, but with different implications. This is where you have to go and see which of them implies something about what it models which you haven't seen before, which if you can then observe it would help you decide which is the better description (
vide the 1919 solar eclipse expeditions).
Where things really get heated is when people start arguing about which metaphor is 'really real', especially when they don't actually go and look at either the mathematical analogy itself, or the phenomenon which it is being used to describe.
I could go on about how the scientific method isn't really what most people take it for, either; not only isn't it a way to find objective truth - an objective truth which, to be fair, the whole system takes as a given for it to work at all - but most modern scientists would say that it
can't be used to find absolute truth, only to refine the accuracy and precision of those descriptions (previously known as 'theories', though the term preferred by most working scientists today is the broader and more neutral 'model') to be a better approximation of that underlying truth. And trust me, scientists are as prone to egotism, mistakes, and bias as anyone - see the works of Stephen Jay Gould for examples, both for his lovely pop-sci descriptions of other scientists' foibles, and for the rather egregious and glaring personality flaws which are reflected in his own professional work.
(I truly adore Gould's books, but he definitely had feet of clay.)
The point is, while both science and mathematics may have started out as grand searches for The Truth, they ended up (like all human endeavors) settling on a pleasing fantasy instead. The fact that this fantasy is, at times, a fair approximation of what appears to be actually happening, is almost happenstance.
(See the works of Karl Popper for more on his assertions about why the existence of an objective physical reality cannot be proven
using scientific techniques themselves, while also being a necessary assumption for the scientific method to make sense at all. To be fair, Popper's Postulates are somewhat controversial, especially the assertion that falsifiabilty is the dividing line between scientific and pseudoscientific, but it is still worth knowing what he has to say.)