Do you really mean it when you say, "As long as it's not hurting anyone"?

So I'm sure we are all more than aware of the phrase, "I don't care as long as it's not hurting anyone", or some variation of it. Now I know what I'm about to say is not some groundbreaking revelation, but I've found that the people who say this (me included) fall on a sort of spectrum of tolerance. On one side you have people who stick strictly to the phrase and genuinely don't care about, or at least make an effort to accept, anything someone could do that doesn't bring harm to another. On the other side you have people who hold very intolerant beliefs and say the phrase for a variety of reasons, to put up a front, genuinely believing some things cause harm, sheer ignorance to the world around them, etc. I've also noticed that (at least with the people around me living in the PNW) most people fall somewhere in between, closer to the tolerant side of things. These people hold the standard progressive/accepting views, but when it comes to less talked about things (mainly things popular more in online spaces) they hold lots of bias towards them. I don't want to give too many examples, but a big one I've picked up on are how people feel towards furries. Being a furry isn't harmful to anyone, it's perfectly safe, doesn't disturb peace, and most furries are pretty chill. Despite this, they still recieve a lot of hate from people claiming to be tolerant. What I want to know is, can you relate and how do you personally feel about your sense of tolerance?