Physics is essential to all engineering disciplines. Engineers are not mathematicians. I think that your idea is good, and you would make a good mathematician.

Or electronic enginering!

Well... I would like you to consult with your teachers and seniors first. As you know, electrical engineering has a variety of branches from electric power systems to semiconductors to digital circuits to embedded systems to optoelectronics. Yes, subjects taught in electrical engineering require a bit strong knowledge in mathematics and physics. However, there are some subjects that require more knowledge in mathematics than physics.

electrical engineering does not have much relation with physics

it has more of maths and calculus which you like

I'm really good at math and love my calculus classes but when it comes to physics, I absolutely loathe it. I'm seriously considering switching my major to mathematics. Would this be a good idea or should I just trudge through physics?