The Germany captain struck after 109 minutes to break Blackburn hearts after Rovers had bounced back from conceding a first-half Frank Lampard opener to set up extra-time through Jason Roberts' first goal for the club.
With the game drifting towards penalties, Ballack was in the right place at the right time to fire home a low shot after Salomon Kalou's close-range effort from Shaun Wright-Phillips' cut back landed at his feet.
Ballack's winner was a bitter pill for Rovers to swallow as they had survived a good start from the pre-match favourites to have the better of the second-half exchanges.
Chelsea though will now meet Sir Alex Ferguson's side at Wembley just four days before another potential meeting in the Champions League final in Athens should both sides successfully overcome semi-final opposition in Europe.
Lampard's 21st goal of the season broke the deadlock and had looked like it would be enough for Chelsea, who relied on good fortune and some excellent goalkeeping from Petr Cech at times.
The England midfielder fired home with a 16th minute angled shot after latching on to a perfectly-weighted Ballack through ball and skipping around Ryan Nelsen before beating Brad Friedel.
Joe Cole returned to the Chelsea starting line-up for the first time since November and could have doubled the advantage in the 22nd minute but drilled a Didier Drogba knock down over the crossbar.
David Dunn then responded in kind for Rovers with a powerful effort that had to be beaten away by Cech for a corner.
That effort seemed to inspire Blackburn and they should have pulled level on the stroke of half-time but Christopher Samba headed Dunn's corner over from point-black range.
After the interval, Mark Hughes' side kept their foot on the gas and after Andriy Shevchenko had missed a sitter when the ball struck his standing leg on the penalty spot, Cech was again the hero as he got down at his near post to divert Roberts' powerful shot behind.
Morten Gamst Pedersen then saw his back-post header from a pinpoint David Bentley cross drift wide before Rovers finally found the back of the net with a 64th minute equaliser.
Roberts gambled and ran in front of John Terry before meeting Pedersen's low free-kick from the right with a deft touch which deflected it beyond the helpless Cech.
But rather than build on their equaliser, it was Chelsea who bounced back and Ballack was inches away with an 80th minute free-kick which whistled past Friedel's upright.
Pedersen wasted a gilt-edged chance as his 12-yard header was saved when unmarked, before Cech palmed away Terry's misdirected header from a long throw as the seconds ticked away.
However, Chelsea dug deep in extra-time and looked most likely to score with Rovers tiring.
And Ballack produced the winner in the second period, before Michael Essien came within a whisker of adding to the heartache for Blackburn with a 30-yard piledriver which cannoned away off the crossbar.