Books and their supposed correlation with femininity?

I've been called feminine for not doing enough sports instead of reading books. And though I have nothing to argue on the behalf of how much sports activities I do, which, quite frankly, isn't a lot. I don't enjoy going all serious with a bunch of people I have the bare minimum of association with. I work out here and there, maybe throwing a ball with people close to me, but that's the end of it.

However, I don't understand the book take. What about the books that I am reading is so feminine? I'd get it if I were reading a young adult romance or fantasy of a sort, but those are genres of books I stray away from most. The majority of books I tend to read are old and so are most likely to be written by men, and so were the readers of the time. 

I have read books written by women; my favorite, for example, is Donna Tartt. I loved Patricia Highsmith's "The Talented Mr. Ripley". In my opinion, both Tartt and Highsmith write such well-thought-out and engaging stories with appealing characters you can't help but dissect every little bit of information given about them; Tom Ripley was a fucking goldmine of analysis for me; the whole back of the book is filled with notes on notes worth of introspection.

But that's basically it for the women; most I read are men. Some might have been homosexual, one or two still are, and some might not have been so masculine or were suspected to have been homosexuals, but Marx wasn't a homosexual. Was he? Even if a writer is, a book about a capitalist degenerate getting more brutal with his victims as time goes on, who are mostly women, isn't the most socially standardized depiction of a feminine book, unless that has changed; update me.

I don't see books as a whole to be necessarily gendered; apart from modern romances and "How to..." books, there really isn't a clear depiction of for what gender the book is written for, at least in modern days. A 16-year-old girl can read "Poor Folk", and a 42-year-old man can read "Mrs. Dalloway" if he pleases; neither of the books has a big fat warning in front saying:

"WOMEN ONLY!" or "MEN ONLY!"—at"least I hope we haven't gotten to that point.

Literature isn't gendered; whether or not you read it or refuse to read it is a matter of literacy. I don't know what about reading makes it feminine, why it is suddenly a feminine trait when for centuries it has been a matter of class and intelligence, which was associated with men due to access to higher education for women being non-existent before the 19th-20th century. Now that education is accessible to a lot more people, literature is a reflection of knowledge and the ability to sit down and pay attention and form opinions about text, a situation, and the world all around.