Striker Daniel Sturridge, who is currently in hospital undergoing tests for suspected viral meningitis, and defender Ryan Bertrand were both selected for Team GB, while Juan Mata and Oriol Romeu are set to join up with Spain.
"As a manager, you would like to have them with you in pre-season, but it is the Olympic Games and players will want to be involved with it," he said. "It is a great opportunity for the players to be involved in the Olympic team, one that the players do not want to lose out on. It is not ideal, but there will be many clubs in the same situation as us."
Di Matteo added: "Pre-season is very important for the players to put the foundation in for the season.
"It is a bit disrupted by the European Championship and the Olympics, and it will be towards the end of July or the beginning of August before we have the whole squad together.
"In that sense it is a little bit disrupted, but we will certainly be working hard during pre-season."
Mata made his only appearance for Spain in the final of Euro 2012, and Di Matteo admitted the player would have to be carefully managed when he does finally integrate back with the Chelsea squad.
He said: "We will have to see with Mata, because he was at the Euro champs as well
"It depends when then go out of the competition and how he feels physically and mentally."
Di Matteo added: "There is always this dilemma in the national teams. There is a friendly in August as well for some, but we will have to work around it."
Source: PA
Source: PA