Love Affair (1963)

When two wives realize that their wealthy and otherwise upstanding husbands have mistresses, they invite the 'other women' to stay in their houses for awhile. Realizing that they are breaking up happy families, the girls eventually agree to give up the married men.