The grandfather, Lomer Johnson, made it out a third floor window and onto the roof before being overcome by fumes.
"The grandfather was found just outside the structure on a small roof covered in debris and inside the window we found one of the children," Stamford Fire Chief Antonio Conte told "Good Morning America." "It appears that the grandfather has one of the children with him, tried to exit the structure but was overcome and passed away. And the little one passed away just inside the window."
Johnson was one of five who died in the blaze. His wife Pauline and their three granddaughters, Lily, 10, and 7-year-old twins Grace and Sarah, were also killed.
The only survivors were the girls' mother Madonna Badger and her friend Michael Borcina, a contractor who had been working on the home. They were able to escape from the first floor of the five-bedroom home. Badger and Borcina were hospitalized with burn injuries.
The three-alarm fire started at about 5 a.m. on Christmas.
Though reports have said the fire was started by the embers of yule logs, Conte said that the official cause of the fire has not yet been handed down from the fire marshal's office.
The $1.7 million Victorian home was torn down Monday after the fire department deemed the destroyed structure unsafe.
Lomer had recently fulfilled a life-long dream of playing Santa at Saks Fifth Avenue, at the urging of his oldest granddaughter. A few years ago, he retired as a safety and security director and then he and his wife moved to the area to be closer to their daughter and grandchildren.
Badger was a fashion consultant and advertising executive, who created the Mark Wahlberg Calvin Klein underwear ads.
Badger climbed onto the roof, desperately trying to break a bedroom window, but the flames had spread too quickly, Stamford Police Sgt. Paul Guzda told ABC News.
Relatives said Badger had been recently divorced and moved from New York City to the affluent suburb of Stamford and renovated the home for her family.
A neighbor said he woke up to the sound of screaming and saw the house engulfed in flames, ABC News New York station WABC-TV reported.
"It was a male voice, and it was just, 'help, help me,'" neighbor Charles Mangano said.
The fire was so large that firefighters struggled and were repeatedly forced back by the flames.
"They made entry to the third floor bedroom and pushed through two rooms, but were unable to find the children and the intense flames pushed them back," Conte said. "They tried a second time to get in and again they were pushed out."
Asked how his firefighters are coping in the aftermath of the tragedy, Conte replied, "Not very well."
"I have 70 firefighters that I've ordered back today for counseling," he said. "They're big, burly guys and to see them break down and the scene was too much to bear. It's really tough."
Psychologists and members of the Fire Department of New York are on hand to help with counseling, Conte said. VIDEO