Things that are offensive

Ok so for a while now I've noticed people critisising things and saying that something is bad because it's offensive and this is something I don't agree with. It happens a lot with comedy where someone will say person made a joke about thing that is offensive to some people therefore the person that made that joke is a bad person, which is really bad logic. I'm not saying that all offensive jokes are good because of free speech or whatever but just that the bad ones are bad not because they're offensive but for what makes them offensive. An example of this would be a racist joke, a racist joke isn't bad because it offends people but it's bad because it's racist which is what makes it offend people, if you say that a racist joke is bad because it offends people then by the same logic a joke making fun of racists would be bad because it would offend racists.