Don't give up your dream.
First, why do you think that you are bad at math? Because of your math grades? Maybe the problem is not you, but the environment (the teacher, the group, etc.).
You ste using math in physics. Is it hard for you to use math to solve physical problems? If you are doing fine with quantitative problems in physics, your math understanding is probably OK.
You can also test your math skills by taking an independent math test, such as SAT or ACT exploratory test. If you score high, talk to your math department leader about it.
Personally, I have never enjoyed math, and I was able to finish my engineering degree with a 4.0 GPA, and I am currently studying astronomy. For some people, math is a passion, for me, it's just a tool. As long as you can use math effectively as a tool you'll be fine.
Another perspective - Einstein wasn't great at math easier. He used help from a friend (Tullio Levi) to develop the Theory of Relativity. Tulio was the guy who found errors in Einstein's math work and was able to put him on the right track.
So again, don't give up on your dream. You may need to work a bit harder, but it does not mean that you won't be able to make it.