This week the Supreme Court will hear arguments over the constitutionality of same-sex marriage, and all eyes are on Justice Anthony Kennedy—as they should be, for the fate of marriage equality likely rests in his hands.

Unlike other cases that implicate personal privacy rights, the marriage equality cases appear before the court framed in part as a conflict between state rights and federal power. We don’t often frame social issues like marriage equality as being only about the balance of power between the federal government and the states. That’s because, aside from the last neo-Confederate holdouts, we’ve largely accepted that a person’s rights should not be defined by that person’s state of residence. In other words, we accept that a citizen of Alabama should be entitled to the same basic rights and protections as a citizen of California. But in both of the the marriage equality cases before the Supreme Court right now, the question is, “Is marriage a fundamental right, and if so, to what degree can states legally restrict that right?”

Framing the fight over marriage equality as a battle over federal and state powers has risks. The greatest of these risks is that it provides an opportunity for the conservative court to again limit the reach of federal power while granting states license to pass and enforce state-level marriage equality bans. Much like the battle over the constitutionality of the Affordable Care Act, the Supreme Court could do so in a way that looks like a win to marriage equality advocates but threatens broader equality rights overall. For Justice Kennedy, the key swing vote in these cases, this may be too good to refuse.

In NFIB v. Sebelius, the decision that upheld the constitutionality of the Affordable Care Act, Chief Justice John Roberts and the rest of the court embraced the legal argument that states have the power to limit the federal government when it came to establishing and administering the requirements of the federal Medicaid program. This was a radical re-understanding of the relationship between the federal government and the states. The law was upheld, but the decision fueled hard right social conservatives in attacking social benefits programs. It was a victory for social justice advocates, but one that has come at a steep cost, as states including Texas and Arizona push for ways to opt out of social benefit programs like Title X.

Marriage equality faces a similar threat. In Hollingsworth v. Perry, the court will hear arguments as to whether Proposition 8, a California voter initiative that defined marriage as a union between a man and a woman, violates the federal Constitution. In U.S. v. Windsor, the court will hear arguments challenging the part of the federal Defense of Marriage Act (DOMA) of 1996 that defines marriage as “only a legal union between one man and one woman as husband and wife” in determining federal benefits.

Should the court defer to the states on the issue of defining marriage and uphold Prop 8 while striking Section 3 of DOMA, then, much like the battle over access to abortion care, the battle over marriage equality will move primarily to the states. And much like the battle over abortion access, it could very well be Justice Kennedy that takes it there.

For Kennedy, the issue of fundamental rights and privacy is a cornerstone in his legacy on the court. In 2003, he authored the opinion that struck state sodomy laws as an unconstitutional violation of the right to privacy in the landmark case Lawrence v. Texas. Eleven years earlier, when the court was asked to overturn Roe v. Wade in Planned Parenthood v. Casey, Justice Kennedy reportedly sided at first with the conservative wing to reverse Roe‘s foundational privacy ruling, but later changed his vote after meeting with Justice Harry Blackmun, the author of the Roe decision, who expressed his concern to Kennedy that history would judge him harshly should he be the vote to overturn Roe. With public opinion squarely in support of marriage equality, it seems likely that Kennedy is  remembering Blackmun’s warning this week.

At least 31 states already have laws that define marriage as the union of one man and one woman. Why wouldn’t Justice Kennedy vote to respect those laws under the guise of reigning in the federal government? It’s just the kind of straw-man reasoning that justified his “support” of privacy rights in Roe, despite his future decisions that have done nothing but undermine that right. Next, he could do so in an opinion that would ultimately side with a same-sex couple.