The Londoners trail title rivals Manchester City by five points in the Barclays Premier League with just two games gone after being thrashed 3-0 at the Etihad Stadium on Sunday.
Ivanovic recognises the winless start and position do not look good but he insists it is far too early in the campaign to be getting concerned.
The Serbian, who endured a difficult game at right-back, said: "At the moment, yes it looks a big gap - two games, five points - but also we have to look at our situation and improve.
"It was a very difficult game for us but we can come back
This is the Premier League
Every game is another final so we have to look forward.
"We know what we haven't done well in the first two games but there is another 36 and we have to improve ourselves
For sure we will not give up after these two bad results for us
Yesterday, we didn't compete with them and this was the problem for us in this game."
Chelsea ensured a troubled build-up to the game after a week of headlines dominated by manager Jose Mourinho's row with two of his medical staff.
This latest result, and its emphatic nature, could increase tensions at Stamford Bridge and Ivanovic concedes there can be no more revelling in last season's title success.
The 31-year-old said: "We know that this is another year
Last year we won the title and we play with extra pressure now and we have to adapt to that
We have to forget last season and start thinking about the points this season."
Chelsea were outplayed in the first half and goalkeeper Asmir Begovic had already saved three times from Sergio Aguero before the Argentinian, who had also shot wide, opened the scoring after 31 minutes.
There was controversy as Chelsea striker Diego Costa appeared to be caught by an elbow from Fernandinho and had to be prevented from confronting the Brazil international at half-time.
But Chelsea were unable to find a way back into the game as Ramires had a goal disallowed for offside and Eden Hazard was denied by Joe Hart in the second half
Vincent Kompany and Fernandinho made the points safe in the last 11 minutes.
Mourinho, who made the surprising decision to substitute captain John Terry at half-time, described the result as "completely fake".
Ivanovic did not go that far but did find positives.
He said: "In some parts of the game we controlled it and looked good, but after that maybe our concentration went down and we let them create some problems for us.
"You have to be on the top of your game and of your opponent to prove yourself and get the points
I hope in the next couple of games we do that.
"Every team is improving but we can be better than last year as well and we have to show that on the pitch."
Source : PA
Source: PA