one of the many things I've noticed in my social life as an autistic person

So for the past 6 years I've been subconsciously mirroring people this builds fake relationships and fake friendships from my experience

Recently I've begun to act like my true self around these people I can pick up things very easily and I have always had a good read on people so with that I've noticed people acting odd an nervous around me when I get close to them they usually say I have a very confident tone that doesn't really tell me much.

it also scares people that I don't forget anything I used to have a very nice friendship with a specific person then they distance themselves ( word for word a conversation we had a year ago )

overall I'm not emotional or logical I prefer to stay in the sweet spot 

also 60% of the time I prefer being non-verbal.

I don't feel lonely or lost I'm just disappointed am I surrounded by insecure people or am I the problem.?