It's great that you don't care about physical appearance.
I'm sure most men wouldn't think twice about a few pounds here and there, especially after a few years and a couple of kids. However, if we're talking 50 pounds or so than that is going to be a problem for a lot of men. Sure, they wouldn't leave their wife, but I'm sure the man would be voicing his displeasure.