I was out of town and away from the computer when Elliot Rodger stabbed his roommates to death and then embarked on a shooting spree that he described as a “war on women.” That means I missed the response to it that took place online, on Twitter, with women sharing their stories of the everyday violence they face.

But this conversation isn’t new; it feels like something we’ve been building for a while, with each new article or incident of violence. We tell our stories, and some men express shock that this happens, that the world is full of threats if you happen to be female, and even more so if you are queer, trans, or a person of color. And like clockwork, too, some people will pop up to respond that it can’t be that bad, that not all men are responsible. Not all men rape. Not all men sexually harass. Not all men are violent. Not all men catcall.

And now, having written this article, I look forward to being told that not all men say “not all men.”

“Not all men” has become a meme, an in-joke among those of us who speak up in public or semi-public about feminist concerns. My favorite version of the meme is a still from the movie Jaws of a shark landing on a small boat; there’s a speech bubble above the shark’s head that reads, of course, “Not all men.”

It’s funny and yet not funny, this meme, because the shark looks ridiculous and yet is actually scary. It plays on the tension between the sheer predictability of the comment, and the fear we have of men’s anger if we are insufficiently nice to them.

In the days since I heard about Rodger’s violent spree, and especially in the last two since my return to regularly scheduled Internet access, I’ve thought a lot about “not all men.” Mostly, I have thought about how grating, how actually painful it can be, when someone pipes up in my mentions or in real life with some version of “not all men.” Because telling ourselves “it’s not all men” is a requirement for continuing to exist and work in a world that increasingly requires our interactions be public, observable. It is a belief we desperately cling to even when we are boiling over with anger and frustration, enough to say something where people who aren’t close to us may overhear.

This week, though, I’m thinking of the conversation with a new friend a couple of years ago when she looked me in the eye and told me she was afraid of men, and I knew exactly what she meant.

Not all men—but enough of them.

I’m thinking of the college friend, a black belt, who only felt safe dating men who were physically smaller than she was.

Not all men.

Just the ones big enough to win a fight.

Because we all do have men in our lives, even if we do not love them or form relationships with them. Most of us do love at least a few of them, at least a family member or a close friend. We have men for bosses, for mentors and teachers. We need to believe that not all of them are simply trying to get close to us for sex. We need to believe that some of them see us as human beings. We cannot actually escape them, and so, like Blanche DuBois, we have to rely on their kindness instead, and hope that it will be enough. That it will not be all men.

We need to believe that not every boss, after we’ve worked at a new job for a couple of weeks, will turn to us and say, “I hired you because you had those seamed stockings on when you came to your job interview,” and then chide us for not flirting enough with customers for better tips.

We need to believe that the older, accomplished man who offers to read over our articles or papers, critique our art, is actually interested in our work because it is good, or he believes it could be. We need to believe the mentorship he offers is genuine.

We need to believe that our neighbor, with whom we chat late at night while walking the dog, will not suddenly raise an eyebrow and start talking about our legs, our hair, our bodies, and that we will not have to mentally calculate how far it is to the door, how many times we have to go out at night alone, how many days are left on our lease.

And these are the better stories. The stories collected on the “When Women Refuse” Tumblr, started by Deanna Zandt, are far more chilling. Stories shared by friends, relatives, neighbors of women who were attacked, some of them killed for refusing men, for leaving them, for denying them something they wanted.

When we walk down the street, we have to believe that the man whose catcall we ignore will not be like the Indianapolis man who chased after three teenage girls in his car and shot out their windows after they “fled his unwanted attention” at a gas station, or else we’d simply never leave the house ever again. We need to believe that if we break up with our boyfriend, he will not track us down and kill us. We need to believe that if we call the police to get a threatening man out of our house, they will not arrest us instead, throw us in jail for the crime of self-defense.

We need to believe this so hard that we delude ourselves into blaming other women for what they face. Because of course “not all men” is not only said by men. It is said by women who say they aren’t feminists because they don’t hate men; it is said by women who consider themselves feminists but think of other women as “too strident.” It was said to me by commenters when I wrote a piece about the repeated encounters I had with angry, shouting, threatening men when I accepted public speaking engagements. It’s not the men, they say—it’s you.

I understand that response, of course. It is necessary to function, this ability to convince ourselves that we have nothing to fear. And yet that response too tells us to shrink ourselves, that we must be nicer, that we must not speak up too loudly, that we must not criticize men, because after all, it’s not all men.

And so when they pop up in my Twitter feed like that damned animatronic shark, I try to laugh rather than scream. I try to just ignore them.

I know what it’s like to look in the eyes of someone I trusted with my darkest secrets, my body, keys to my house, and to see nothing there that I knew, to be terrified of what he might do. I know what it’s like to try to trust someone again after that, and to have the trying feel too big and too scary and too much. I know what it’s like to throw caution to the wind and trust anyway.

It can’t be all men, can it?