At the Nottoway Plantation in White Castle, Louisiana, an eerie disproportion defines the atmosphere. In the “big house” and in the mansion’s wings, spaces grand and cozy are hung with portraits of pale people in 19th-century dress. A cluster of blonde girls graces a painting in an opulent music room, replete with a golden harp, a rosewood piano, and two other dainty keyboard instruments of the era. Take the long walk to the “boys’ wing” of the sprawling home, and a young, auburn-haired master assumes a jaunty pose in an oil painting above a fireplace.

Even if you’re not taken to believing in ghosts and spirits, a trip to Louisiana can change that for you. And in the big house at Nottoway, the ghosts one feels whooshing about are less likely to be those whose visages stare out from the paintings; they are the ghosts of people rendered faceless by those who recorded the plantation’s history, the ghosts of 155 enslaved African Americans who served the 13 members of the Randolph family (Nottoway’s owners), and who built the mansion and worked the land.

When news broke that Ani DiFranco, the acclaimed feminist poet and songwriter, was under Twitter siege for having scheduled a “Righteous Retreat” at Nottoway for writers and artists, it took me a while to recall it as the very place I had stayed in 2005, two weeks after New Orleans was inundated by Hurricane Katrina and nearly strangled to death by the callous negligence of the Bush administration. So deeply disturbing was my Nottoway stay, and the circumstances that brought me there, that I had locked it away in the recesses of my memory.

In the days just after Hurricane Katrina, New Orleans proper was closed to anyone other than those who were locking it down or cleaning up debris. I wound up at Nottoway with two colleagues from the labor union for which I worked at the time because the hotels in Baton Rouge, where Katrina humanitarian aid operations were staged, were full, and the best the travel experts could come up with was the honeymoon suite in an out-of-the-way plantation. We had come to Louisiana to find out where the members of the union’s locals had dispersed to after the storm, and to assess the working conditions of those involved in the clean-up and post-hurricane security operations.

It was either fitting or ironic—I’m still not sure which—to find ourselves in the ancestral home of the man who once maintained one of the largest workforces of enslaved people in Louisiana’s history, lured, as we were, by a catastrophe in which our nation’s brutal racial history played no small part. Who knows how many of the drowned descended from the people once held here?

New Orleans is DiFranco’s adopted home, which made her choice of a retreat venue all the more puzzling, since it would presumably take a willful disregard of the history of that most historic of cities, once the center of the U.S. slave trade, to not have a clearer sense of the still-raw wounds of what polite white people once referred to as “our peculiar institution.” But for those of us who are white, it is all too easy to minimize the part of our history that makes us uncomfortable.

Have a look at the horrors endured by enslaved women, and the continued effects of those horrors on the lives of women who descended from them, as well as American culture writ large, and one would think that the study of the institution of slavery and its aftermath would be a primary path of inquiry for U.S. feminists. But it is not—at least not for those of us who are white.

It’s hard to imagine the notion of patriarchy more baldly expressed than by the slave-holding culture of pre-Civil War America, where a plantation’s master, usually married, used for his pleasure the women he held as slaves, and then enslaved any children the union produced. Need we look any further for the roots of American rape culture?

Yet despite the obvious pain of this history, not to mention the myriad ways in which it has distorted cultural and sexual relations, we look away. How else to explain the utter shock expressed by DiFranco in the defensive screed, described by Brittney Cooper as a “faux-pology,” with which she announced the cancellation of her “righteous” plantation retreat?

Meanwhile, in the comments section of DiFranco’s Facebook page, to which she subsequently posted a more apologetic note, we find some of her supporters telling her she had done nothing wrong because, they say, her intentions were good.

Now, if you’re a white feminist, you may find yourself a bit bowled over by the push-back on Twitter and in social media by African-American feminists and women of color over what may seem to us like innocent oversights. We expect compassion for “not having meant it that way,” whether the “it” is omitting a visual image of African-American women bloggers from one’s presentation on the feminist blogosphere, comparing the murder of Trayvon Martin to the peril of being a white woman in a sexist society, or, in DiFranco’s case, going along with a promoter’s venue choice that purportedly gave her pause (“whoa”).

But it’s precisely the fact that “we didn’t mean it that way” that is the problem. That we continue to make these sorts of off-handed insults and worse with no understanding of why they hurt—that takes a kind of willful indifference, and a denial of our own history, our history as a nation, as a movement, as American women. It’s the kind of denial afforded by our privilege. There’s just no shaking that.

Remember the television images of people, even babies, begging for help, thirsty and hungry, at the New Orleans convention center? I doubt that an African-American woman who found herself housed at Nottoway in the wake of Hurricane Katrina would have forgotten the name of the plantation eight years later, would have been unsure if that was the same place. But I did. I couldn’t wait to forget it.

It’s not that some irrational and sudden fury over incidents such as the DiFranco debacle suddenly appeared. What you witness today on Twitter, with hashtags such as Mikki Kendall’s #solidarityisforwhitewomen, is the culminated frustration with centuries of disregard of the opinions, contributions and experiences of women of color, especially Black women. But today feminists of color now have, in social media, the kind of platform that was never available to them in the multitudinous spaces, from the academy to pop journalism, where they’ve traditionally been denied anything other than token representation (if that).

As for Nottoway Plantation, allow me to skip the parts of the history already told in the pushback on DiFranco: you can read them here, distilled by Professor Pink of Jezebel’s GroupThink, including the way in which the owners of Nottoway describe the life of the antebellum enslaved population on the estate, and the supposed benevolence of John Hampden Randolph, the plantation master, to his enslaved workers. I’ll simply add that, in the index of his papers, housed at Louisiana State University and listed on its website, is two sets of speech notes “concerning the White League.” No hint is given as to whether Randolph was part of or sympathized with the White League, a sort of vigilante militia in the service of white landowners after the Civil War that specialized in killing Black people who demanded their political rights, and in otherwise terrorizing Republicans, then the party of Emancipation and Reconstruction.

Perhaps this bit of the Randolph family history sheds some light: Nottoway’s own website has a brief entry on one of the youngest of the master’s sons, Peter Everett Randolph, who, the present-day overseers of Nottoway tell us, “fell in love with Alice Thompson, the beautiful mulatto daughter of a former slave, Eliza Thompson.” (Yes, they use the term “mulatto.” And just how Alice Thompson, an enslaved woman, came to give birth to a daughter of mixed race is not explained.) The two ran off to New Orleans, according to the website, and had two daughters. Peter, we’re told, was “written out of the family will for quite some time.”

But a 2005 Masters’ thesis by one Christian Williamson, who participated in an exhumation project of a Randolph family cemetery plot in Bayou Goula, Louisiana, notes the slavemaster’s fury at his son. Writes Williamson:

Because of the ongoing relationship between Peter and Alice, John Randolph eventually disowned his son, destroying any record of him at Nottoway. Allegedly, John Randolph went so far as to tear out Peter’s name from the family bible. According to Mr. [Anthony] Reffells (personal communication 2004), Peter moved to New Orleans and lived in a house officially owned by his mother that was near Alice and their children. [Reffells is described as “a confirmed descendent of Peter Everett Randolph”.]

It wasn’t until his father died that Peter was written back into the family, Williamson writes. On the Nottoway website, richly illustrated with photos of the Randolph offspring, the reader is simply told that no images exist of Peter Everett Randolph. Nor, apparently, are there any of his daughters, or of Alice Thompson, the mother of his children.

Yet the Nottoway website tells us that the descendants of Peter Everett Randolph’s daughters “have come to Nottoway over the years to learn more about their heritage,” suggesting that the true history of America isn’t lost to us; it’s hiding in plain view. We just have to want to see it. Until we do, we’re only talking the talk.