Less than 24 hours after Hilary Rosen made a comment about Ann Romney's history as a stay-at-home mother, President Barack Obama joined other political heavyweights in taking a swing at the Democratic strategist's controversial remarks.

Obama, reflecting on his own mother and his wife, said there's "no tougher job than being a mom."

"Anybody who would argue otherwise, I think, probably, needs to rethink their statement," the president said in an interview with CNN affiliate KCRG, a television station in Cedar Rapids.

His comments came after Rosen, also a CNN contributor, sparked an uproar Wednesday night by saying Ann Romney had "never worked a day in her life" and could not understand the struggles of working moms.

Republicans and Democrats--including members of Obama's re-election team--quickly rushed to the defense of Ms. Romney, calling Rosen out for her remarks.

In an apology the following day, Rosen said she used a poor choice of words and meant to say the Romneys were out of touch with working class families due to their wealth.

The president, however, said families should be "off limits" in an election season.

"I don't have a lot of patience for commentary about the spouses of political candidates," Obama said. "Those of us who are in the public life, we're fair game. Our families are civilians."

While he's never met Ann Romney, he argued she hadn't "necessarily volunteered for this job," adding further that spouses do not deserve to fall in the crosshairs of political rhetoric.

"Me and Governor Romney are going to have more than enough to argue about during the course of this campaign," he said.