The European champions were all over the place at the back in Wednesday night's dramatic Capital One Cup win over Manchester United and were hardly any better in Sunday's first league defeat of the season against the same opponents.
"He's obviously been an influential player for us, for sure, and a fantastic defender," Di Matteo said. "When he plays, he's always played very well for us. The other players have done a fantastic job by coming in and playing when they've been in the team but, certainly, JT is a player that we miss."
The Blues have now conceded 11 goals in just four games since Terry began his domestic racism ban, although the Chelsea captain did play in one of them when he started their Champions League defeat at Shakhtar Donetsk.
Terry serves the final instalment of his ban against the Swans, while manager Di Matteo revealed fellow defender David Luiz would almost certainly miss the game though injury.
That would mean Branislav Ivanovic being pressed into centre-back service in a new-look defence that would almost certainly see right-back Cesar Azpilicueta handed his full league debut, with Ashley Cole returning from injury on the opposite flank.
Di Matteo added: "The last few games, we've conceded a few more goals than at the start of the season.
"Generally, it's been down probably to a few individual mistakes. But I think our track record is very good and, surely, will continue to be."
If not, Chelsea can surely forget about staying top of the Premier League and reaching the Champions League knockout stages.
"We have to improve but I don't think it's just as simple as that," Di Matteo said. "You look at other teams and they're conceding goals, so we certainly have to improve, from the last few games, our defensive organisation."
Source: PA
Source: PA