When most people marry, there is a mutual understanding (usually in the form of a vow) that certain connections outside the marriage relationship will be out of bounds. This includes both physical and emotional connections. When one spouse violates these boundaries, they break their promise and betray their spouse.

Once you become intimately involved with another person (whether emotionally or physically or both... whether one time or many times... whether for a short time or a long time), you have done damage to the underlying trust in your marriage. It is an affair.