Ben Affleck Was Hesitant with Jennifer Lopez Romance Because of 'Responsibility' to His Kids

Ben Affleck got candid about his personal life in an eye-opening interview with longtime radio show host Howard Stern.

The Gone Girl star 49, stopped by The Howard Stern Show to chat about a myriad topics, including his career, reputation, marriage to Jennifer Garner, struggles with alcohol, and his relationship with Jennifer Lopez — including why he was hesitant about rekindling their romance 17 years after calling off their engagement in 2004.

"Was there a hesitation on your part, because now you're a dad?" Stern asked Affleck, noting that Lopez "brings even more attention" to everything he does.

"It crossed my mind for sure," Affleck told Stern of getting back together with the "Let's Get Loud" singer. "My responsibility to my children is the highest responsibility. I don't want to do anything that is painful or destructive to them if I can help it."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Affleck shares daughters Violet, 16, and Seraphina, 12, plus son Samuel, 9, with ex-wife Garner, 49.

For more on Ben Affleck and other top stories, listen below to our daily podcast on PEOPLE Every Day.

"That being said, I know that my life affects them," Affleck continued, noting that the fact that he and Garner are celebrities is a "cross to bear" for their children.

In addition to briefly touching on his romance with the "On My Way" singer, 52, the Tender Bar star spoke openly about his marriage to Garner, and how the couple tried to make their relationship work.

RELATED VIDEO: Ben Affleck and Jennifer Lopez Are All Smiles as They Cuddle Up Courtside at Lakers Game

"The truth was we took our time, we made the decision, we grew apart," he explained to Stern. "We had a marriage that didn't work. This happens … We tried because we had kids. We did not want it to be the model of marriage for the kids. We did our best. I knew she was a good mom, and I hoped she knew I was a good dad."

Although Garner and Affleck finalized their divorce in 2018, the couple continue to amicably co-parent their three children — Garner even joined her ex-husband and Lopez over Halloween to take their kids trick-or-treating together.

In November, a source told PEOPLE, "They are doing really well. Jennifer loves being with Ben. She feels it's truly meant to be."

A separate source revealed that although the couple's schedules are demanding, Affleck and Lopez are looking forward to spending the holidays together this year. "For Jennifer, Christmas is all about the kids," the source shared. "She makes it very special for them. She loves Christmas."

The source added, "This year, it will take some more planning since Ben wants to be with his kids, too. They haven't made any specific holiday decisions yet and are still working it all out."