It matters significantly if you want the relationship to work. The separation of anything emotionally meaningful in sex and physical pleasure is complete if no one cares how many partners the other has had. Sex is then about taking and not giving. It can be something about giving — this is something I agree with the religious types on. It should be that way or else it is empty and completely meaningless. If that’s what you want, fine, I guess you can do that.
Whores do that; that’s how they can cope with the fact they are whores.