After the largest gain of seats for the Democratic Party since Watergate, Democrats seem poised to again select Rep. Nancy Pelosi (D-CA) as the next speaker of the house. But a small coalition of Democrats disagree. On Monday, 16 incoming and incumbent members of the U.S. House of Representatives released a letter declaring they will not support Pelosi as speaker.

In the months leading up to the 2018 midterm election, some Democratic candidates—including a number of progressives—called for a change in leadership. Pelosi has faced criticism for her refusal to endorse policies like single-payer health care or abortion rights as a litmus test for Democrats. But contrary to the narrative that elected progressives are pushing back against Pelosi serving as speaker, much of the current challenge to her leadership stems from center and center-right members of the party.

Of the 16 who signed the letter opposing Pelosi, at least ten are associated with conservative or center-leaning groups: Six representatives or incoming members of the House are associated with Blue Dog Democrats, a coalition of conservative-leaning Democrats, while a number of the representatives are part of the New Democrat Coalition, centrists that advocate for “fiscally responsible policies.”

Reps. Anthony Brindisi (D-NY), Jim Cooper (D-TN), Max Rose (D-NY), Kurt Schrader (D-OR), Jeff Van Drew (D-NJ), and Filemon Vela Jr. (D-TX) are members of the Blue Dog Caucus or were endorsed by its PAC. Of those who aren’t associated with the Blue Dogs, Reps. Kathleen Rice (D-NY), Bill Foster (D-IL), Ed Perlmutter (D-CO), and Seth Moulton (D-MA) are New Democrats.

“There’s plenty of really competent females that we can replace [Pelosi] with,” Rep. Tim Ryan (D-OH), who signed the letter, told the New York Times last Wednesday. In 2016, Ryan led a failed bid to oust Pelosi as minority leader.

One of the names Ryan suggested was Rep. Marcia Fudge (D-OH), the former chair of the Congressional Black Caucus. Fudge—who did not sign Monday’s letter— has not said whether she will run, but has indicated it is a strong possibility. She met with Pelosi on Friday, but said she would not run if Pelosi promised to serve only one more term. “People are asking me to do it, and I am thinking about it,” she told Cleveland.com last Friday.

While Fudge is a member of the Congressional Progressive Caucus, her record contains some choices that are out of step with progressives. In the last congressional session, she was one of two Democrats that refused to co-sponsor the Equality Act, which if passed, would amend the Civil Rights Act of 1964 to prohibit civil rights discrimination against sexual orientation and gender identity. Her office released a statement on Thursday saying she supports the LGBTQ community but didn’t support updating the current Civil Rights Act.

“I want us to do a new and modern civil rights bill that protects the LGBTQ community and updates protections for this era,” she said. “I do not believe it is appropriate to open and relitigate the current Civil Rights Act.”

Pelosi currently holds a 17 percent approval rating among the general public, according to recent polling by Monmouth University. Among Democrats, 23 percent back Pelosi, 36 percent would prefer someone else as speaker, and 41 percent have no opinion.

Prominent Congressional Black Caucus members and Congressional Progressive Caucus members have vocally refused to break ranks. Rep. Karen Bass (D-CA), vice chair of the Congressional Black Caucus and someone originally suggested by Ryan as a contender for speaker, announced her support for Pelosi last Thursday. Civil Rights Leader Rep. John Lewis (D-GA)  and Rep Jim Clyburn (D-SC), two prominent members of the Congressional Black Caucus, have actively voiced their support for Pelosi. And Rep. Pramila Jayapal (D-WA), likely the next co-chair of the Congressional Progressive Caucus, endorsed Pelosi after she strongly committed to providing progressive members more leadership roles.

“Right now, out of the field, I would say that she is the most progressive candidate,” freshman representative Alexandria Ocasio-Cortez (D-NY) said of Pelosi during an Instagram Live stream. “All of the rebellion for the speakership are challenges to her right, and so I think it’s important to communicate that.”

Pelosi needs a majority of the House, traditionally 218 votes, to keep her role as speaker. In addition to the 16 who signed the letter, newly elected members Abigail Spanberger (D-VA) and Gil Cisneros (D-CA) are both on the record saying they will not support Pelosi. Other newly elected members like Andy Kim (D-NJ), who initially said he’d oppose Pelosi for leadership, have stayed silent since their elections.

Democrats are scheduled to meet behind closed doors on November 28 to pick their leadership. The full chamber will elect the next speaker on January 3.

While more progressive members may not be challenging Pelosi for the role of speaker, they are making internal moves to fortify their position: Rep. Barbara Lee (D-CA), co-chair of the Pro-Choice Caucus, is a strong contender for Democratic Caucus Chair, the fourth most powerful leadership position in the party (previously held by Rep. Joe Crowley). And Ocasio-Cortez, who defeated Crowley in his Democratic primary, recently backed Justice Democrats’ campaign to primary other incumbent Democrats who refuse to push back against money in politics and or don’t represent the communities they come from.

Waleed Shahid, communications director for Justice Democrats, said that Pelosi has not endorsed Medicare For All, free college, taxing Wall Street transactions, or ending private prison and detention facilities. But that doesn’t mean that Justice Democrats, a progressive political action committee that backed Ocasio-Cortez in her primary, supports the current challenge to Pelosi.

“We are fighting to create a Democratic Party that is mission-driven and fights for firm principles, not a party that seeks bipartisan compromises with a Republican Party that constantly lies in order to divide and loot the people,” Shahid told Rewire.News.

“We don’t support the corporate-backed Blue Dogs’ maneuvering for Speaker.”