Everyone who has seen the Lord of the Rings or The Hobbit trilogy knows that Gandalf is an unstoppable figure. He manages to persuade two of the most stubborn men in Middle Earth (the sick and poisoned mind of King Theoden of Rohan, and the paranoid and delusional Steward of Gondor Denethor) to let him intervene in their affairs, even when all of the treachery and darkness around them told them not to. Across both sets of books, Gandalf battles many enemies, slays many orcs, protects (and sadly loses) many friends, and he is the instigator of the two quests which, each in their own way, determine the fate of the entire world and all of its people.
This is even more impressive in The Hobbit and the Fellowship of the Ring than it is in the Two Towers and the Return of the King, because in these earlier stories he is still Gandalf the gray. Even at this point, before he has come into the most powerful version of himself, he is capable of great good, but also great evil, and holds the future of the world in his hands. During the famous battle against the Balrog in Moria, Gandalf manages to save his friends by distracting the beast, and breaking the bridge of Khazad-Dum so that the creature cannot follow the others. He gives his life to defend his companions, and he manages to beat this massive force of darkness and dread. This has led many fans to question why Gandalf didn’t simply go into the halls of Erebor during Bilbo’s quest 60 years earlier and get rid of the dragon himself?
If the whole point of the mission to reclaim the lonely mountain was a political strategy to fortify the north and prevent the Necromancer (who is later revealed to be Sauron) from using the dragon against his enemies, surely it would have been simpler for Gandalf to defeat Smaug and save the dwarves the effort. It also could possibly have saved hundreds of lives of the people of Lake Town, who were literally caught in the crossfire between the dragon and Thorin’s men. However, there are many reasons why Gandalf dealing with Smaug wouldn’t have worked.
First of all, although Gandalf does manage to defeat the Balrog in Moria 60 years later, the Balrog is a very different enemy to a dragon. This is a creature of fire and smoke, the same as Smaug, but a Balrog’s power comes from its dark and evil spirit, an ancient magical strength that is more akin to that of a maiar or Sauron himself.
When Gandalf battled the Balrog, it was a battle of will and of magical quality, rather than a battle of physical strength as it would be in the case of Smaug. A dragon is a much more tangible bodily enemy, with impenetrable scales, the ability to breathe fire, and most importantly wings. If Smaug had simply flown above Gandalf and rained fire down on him, there’s no saying if the wizard would be able to survive.
Although both the Balrog and Smaug are creatures of fire, and Gandalf is able to match them thanks to the power of his fire ring, defeating the Balrog takes far more spiritual prowess, which is why Gandalf is able to essentially ‘level up’ and comes back as Gandalf the White, whereas defeating such an immense physical enemy as Smaug would have fed into more of Gandalf's weaknesses in his physical limitations than his strengths in spiritual and magical abilities.
The second reason to mention is that the maiar aren’t typically supposed to insert themselves into the problems of men. Both Saruman the white and Radagast the brown are much better at this during The Hobbit than Gandalf is, and generally prefer to keep to themselves, and watch events from a distance. They were put on Middle Earth to guide and help the peoples of the world, rather than to take over and solve their problems for them.
This is why, in both Bilbo’s and Frodo’s quests, Gandalf is more of an instigator than a direct participator, only stepping into a situation when all other hope is lost, and a dire situation that might destroy everything, calls for it. In this way, Gandalf’s main purpose was to help Thorin and his dwarves find their own paths, and meet their own destinies. In fact, in the past, when Maiar beings got involved in events that they shouldn’t have, the damage was so colossal that it often created more harm than good.
So if Gandalf had battled Smaug directly, it may have resulted in more tragedies than Lake Town, and might have wiped out Thranduil's kingdom in Mirkwood, as well as many other nations nearby. Even if Gandalf could have defeated the dragon, it was not his place or responsibility to do so.
The third reason is that, in taking out Thorin’s involvement with the dragon, Gandalf would have prevented the strong and necessary bonds from being created that came in to help in the War of the Ring 60 years later. During the quest, the dwarves develop a strong relationship with the survivors of Lake Town, including Bard, whose son Bain (the boy that helps him shoot the Black Arrow in The Hobbit film) helps Aragorn and the others fight in the battle against Sauron during Frodo’s quest.
The men around the area are instrumental in rebuilding Dale and helping the Iron dwarves set up a stronger kingdom. They also establish strong connections with the elves, which is probably why the elves, the dwarves, and the race of men are all called to the Council of Elrond all those years later to deal with the ring of power.
Without these relationships being formed now, the War of the Ring would almost certainly have gone differently, and in some ways, Gandalf knew all of this when he made the decision to keep out of the reclaiming of the Lonely Mountain.