Forty-seven years ago, Gillian Relf gave birth to Stephen, who was diagnosed with Down syndrome shortly afterward. Last month, in a story in the UK’s Daily Mail, Relf wrote that while she loves her son, she wishes she had aborted him. For parents of fetuses with Down syndrome, Relf implied, abortion is the only way out of a lifetime of obstacles.

The broader Down syndrome community of parents typically responds to stories like these with outrage or sadness. Instead of acknowledging the validity of the hardship narrative, we often choose to promote superficial cute and happy imageries of life with Down syndrome, as if to acknowledge the challenges would be to promote aborting fetuses with disabilities. We, too, have bought into a narrative of limited, inaccurate choices: that someone must either constantly celebrate Down syndrome with no discussion of the difficulties it can present, or effectively be supporting eugenics.

So here we have two different false binaries. Neither is true; both remove the agency from people with Down syndrome. Furthermore, neither identifies the clear missing piece—what if, instead of leaving people like Relf isolated and struggling, we identify ways to build a robustly inclusive and caring society? What if we fight to expand access to support for all parents? We can break these dualistic fallacies apart.

For some people, abortion may be the right decision. For others, the choice to terminate a fetus with Down syndrome may reflect a fundamental lack of understanding about disability. In all cases, we need to unpack the way we construct these conversations, rather than criticizing individual decisions.

As the prenatal testing landscape changes, the debate over abortion and Down syndrome has become increasingly fraught. In Gillian Relf’s era—in fact, until very recently—accurately determining whether a fetus had Down syndrome required an invasive second-trimester amniocentesis or chorionic villus sampling. Today, new non-invasive blood tests can locate fetal blood cells in a pregnant woman’s blood stream much earlier, with a lower risk of false positives. The new testing regimes raise complex ethical questions. It’s arguably different to say “I don’t want any baby” as opposed to “I don’t want this baby because of a prenatal diagnosis.” The latter feels more personal and raises the specter of eugenics.

The piece in the Daily Mail is not unusual, nor is the backlash it generated. As a father of a boy with Down syndrome and a journalist on disability issues, I hear these kinds of stories every time I write about the fraught issues of prenatal testing, abortion, and disability. Our life is challenging but rewarding, and I reflect both of these facts in my writing. Too many responses, however, either maintain that any discussion of abortion is tantamount to genocide, or revert to the casual ableism of “I would certainly terminate a Down pregnancy myself, and I wouldn’t waste much time thinking about it.”

To get beyond these conceptual traps requires looking past the abortion issue, as hard as that can be, and instead rethink how society might function better for families like Relf’s and mine. Relf talked about the deep isolation she has felt throughout her life as Stephen’s mother, driven in part by her inability to cope with Stephen’s “sit-down” protests. She’s afraid that he has been physically abused by other caregivers. Perhaps worst, she has the general sense that Stephen’s life is empty of meaningful relationships or activities. These are real concerns, and even people who are appalled by her rhetoric should not dismiss them.

It’s true that Relf might have avoided many of these challenges through an abortion. Of course, that’s not the only answer. Progressive social policies, matched with a change of attitude, can and have made huge differences in the lives of both people with disabilities and their caregivers. Society is already much more inclusive than during the days of Stephen’s childhood, but there’s still plenty of room to work on the isolation that so many families fear.

Since 1990 in the United States, with the passage of Individuals With Disabilities Education Act (IDEA), schools have normalized inclusive education. That means that everyone younger than 30 or so likely grew up with kids with disabilities in their classroom. The American With Disabilities Act (ADA), passed the same year, mandated that more programs and public spaces be adapted for people with disabilities as well.  People still stare when our children act “abnormally,” such as with Stephen’s “sit-down protests.” Still, the awareness campaigns promoted by individual activists and groups such as The Arc, the Autistic Self Advocacy Network, and the National Down Syndrome Society have begun to create a more inclusive world. Attitudes are changing, albeit slowly. And thanks to better therapies and new medicines, life expectancies and general health are also improving (at least for better-off people).

But there’s still such a long way to go before we can tell families, caregivers, and expectant parents that if they choose to carry a pregnancy to term, they and their new child won’t be alone. Sadly, in this age of austerity, policies are moving away from providing the kinds of support parents need. In the United States, funding for respite care programs, which are designed precisely to ease the kind of intense caregiving work that Relf discussed, is being cut or reduced. Meanwhile, the United Kingdom, historically much better for people with disabilities, is now run by a government drastically slashing disability supports in the name of budget cuts.

The situation looks even more grim for adults with developmental disabilities. At least IDEA provides for a clear structure for the lives of children with disabilities, as well as broad social contact. Once adults turn 22 (or 19 in some areas), they “fall off the cliff.” They end up spending their days stuck in their parents’ homes, without community support or a way to maintain their skills. People whose disabilities and life circumstances—in terms of access to transit or the ability to find jobs—allow them to work are still too easily confined to sheltered workshops and paid pennies per hour, though thankfully that model is being phased out.

Meanwhile, people with disabilities and their caregivers have trouble saving money, because our social security system requires keeping people poor in order to qualify. It’s not legal for people receiving supplemental security payments to save more than $2,000—but those payments aren’t actually enough for most recipients to live on.

And our current legislature isn’t improving the situation. Right now, Congress has an act before it called Achieving a Better Life Experience Act (ABLE) that would enable the creation of special savings accounts. Under ABLE, some people with disabilities could work, place up to $100,000 in these tax-free “529” accounts, and use those funds for medical care, education, training, assistive technology, housing, and other essential needs. With ABLE, many people with disabilities would be able to achieve a greater degree of independence. Legislators from both parties have sponsored it, and the president has promised to sign it.

In a more functional Congress, it would have been passed a year or more ago. Now, it’s being threatened by the Heritage Foundation as too expensive, and it may not make it out of the current Congress at all. This program isn’t even a new benefit, just the opportunity to work and save—and even that may be too much for the U.S. government to enact.

Every story about the lives of people with Down syndrome, the choices parents and expectant parents make, and the whole discourse of abortion take place in the complicated context of these types of issues. We need to push back against the binary of hardship vs. abortion, but without erasing the very real difficulties that so many parents feel in our society. Instead, let’s get to work doing what we can to solve the hardships, while making sure every parent is free to make their own choices.