I agree, it's not fair. Even in situations where the unfaithful spouse did more to initiate and drive the affair, people tend to judge the "other" as the marriage-breaker. You may hear some rather unkind things said to you or about you. There's not much you can do about that. In fact, trying to stamp out the rumors just helps them spread, and arguing your case rarely wins over supporters. You need to give attention to yourself and be honest about what you've been through. Find support where you can...even if it's a single friend or counselor. Listen to the voices that care enough about you to love you and tell you the truth. Learn to ignore the rest.