"He went out and just never came back," Michael Carroll said their mother told them, Newsday reported.
There was no missing person report filed and no search conducted. Dorothy Carroll died in 1998.
Although it's unclear why, members of the family believed their father could have been buried in the home, Suffolk County police told WNBC.
"It was just always a family legend or lore that the father was buried in the basement since 1961," Suffolk County Police Detective Lt. Kevin Beyrer told WCBS.
Michael Carroll bought the house from his mother in the 1980s. The Carroll family had owned the house since 1955, WCBS reported. The three-bedroom home was originally built in 1925.
Through the years psychics and paranormal investigators visited the site.
A paranormal investigator felt an energy at the home during a June visit. While skeptical, Michael admits strange events had occurred there, Newsday reported. A visitor said they saw a man's shadow. Other times lights had come on inexplicably.
Michael started digging in the basement a few years ago. He stopped out of fear of causing structural damage to the house, Newsday reported.
But he had to keep going. A company with a ground penetrating radar indicated there was a disturbance in the soil, about 5 or 6 feet deep.
His sons, Christopher and Michael Jr., started digging a few months ago. It was difficult work. They cut through a concrete wall that surrounded an old well. They dug until about 10 p.m. Tuesday, when they found what appeared to be human bones, Newsday reported.
Suffolk County police are now investigating. They will use DNA and any other medical records available to determine if it is indeed George Carroll. They could not determine a cause of death, Newsday reported.
If it is George Carroll the family hopes to give him a proper burial at Calverton National Cemetery.
"We felt abandoned as kids," Steven Carroll, now 61, told WNBC. "But he was here the whole time."