Couple that saved over $1 million to quit work in their 30s say this is the hardest part about retirement