if I have a more powerful motor, will my ESC over heat?
Certainly can. Remember, too, that what we're talking about above is all brushless stuff; your current set up is brushed. Some brushless ESCs can drive brushed motors, but a brushed ESC and a brushless motor does not work.
As for power levels, Godspeed touched on it, but I'll expound a little.
Lower turn = more revolutions per volt. Brushless motors are often rated in KV rather than turns, although often both are listed. So, for example, in brushless, a 17.5T 540 motor is about 2,200 KV, or 2,200 RPM per volt. A 5.5 turn will come in around 5,500-6,000 KV. But lower turn/higher KV motors also cannot take as much voltage (due to max RPM ratings), and do not generate as much torque. There's a whole lot more to it, but this should help you get started in selecting parts.
I will say that I have come to prefer KV rating to turn rating. Reason being, motors of different sizes and even manufacture may have the same turn rating, but radically different KV. Take a 12 turn motor; in 540 size, that's around 3,200 KV. But in a 280 size can, it'll work out to about 7,500 KV. Conversely, in a large 40-42mm 1/8 scale motor, 12T will only be about 1,800 KV. And two motors of the same physical size and turn rating may vary quite a bit in RPM/volt as well. As such, calculating gear ratios is easier with KV ratings.
Personally, I like going with middle of the road in most applications. Most of my 1/10 scale critters are running motors between 3,000-4,000 KV. My 1/8 scales get 1,700-2,200 KV cans. In 1/16, I use 4,000-5,000 KV motors. I find it gives one more flexibility. I can gear higher for speed runs, but geared more moderately, it's not taxing everything as hard as a lower turn motor would.
Whatever you go with, just pay attention to ratings. A lot of ESCs specify a KV or turn rating for a given scale on a given surface for a given voltage. The specs might read something like:
2S LiPo/6 cell NiMH: >8.5T on road/>10.5T off road
3S LiPo/9 cell NiMH: >12T on road/>17 turn off road