WASHINGTON (AP) — Advocates for closing the Guantanamo Bay detention center were optimistic when President Joe Biden took office. And they were relieved this summer after the U.S. released a prisoner for the first time in years. Many are now increasingly impatient.

In the months since that release, there have been few signs of progress in closing the notorious offshore prison on the U.S. base in Cuba. That has led to increased skepticism about Biden’s approach as the administration completes its first year and the detention center reaches a milestone Tuesday — the 20th anniversary of the first prisoners’ arrival.

“President Biden has stated his intention to close Guantanamo as a matter of policy but has not taken substantial steps toward closure,” said Wells Dixon, an attorney with the New York-based Center for Constitutional Rights, which has long taken a leading role in challenging the indefinite confinement without charge at the base.

“There’s a lot of impatience and a lot of frustration among advocates and people who have been watching this,” said Daphne Eviatar, director of the security with the human rights program at Amnesty International USA.

Without a more concerted effort, those who want the center to close fear a repeat of what happened under President Barack Obama. Obama made closing Guantanamo a signature issue from his first days in office, but managed only to shrink it in the face of political opposition in Congress.

“We can’t forget what this country did 20 years ago and is continuing to do today,” Eviatar said. “This administration has a lot on its plate, certainly, but this is such an egregious human rights offense.”

There are 39 prisoners left. It’s the fewest since the detention center’s earliest days, when the initial groups, suspected of having a connection to al-Qaida or the Taliban, arrived on flights from Afghanistan — hooded, shackled and clad in orange jumpsuits — to what at the time was a sleepy U.S. outpost on the southeastern coast of Cuba.

Guantanamo became the focus of international outrage because of the mistreatment and torture of prisoners and the U.S. insistence that it could hold men indefinitely without charge for the duration of a war against al-Qaida that seemingly has no end. The critics grew to include Michael Lehnert, a now retired Marine Corps major general who was tasked with opening the detention center but came to believe that holding mostly low-level fighters without charge was counter to American values and interests.

“To me, the existence of Guantanamo is anathema to everything that we represent, and it needs to be closed for that reason,” Lehnert said.

At its peak, in 2003, the detention center held nearly 680 prisoners. President George W. Bush released more than 500 and Obama freed 197 before time ran out on his effort to whittle down the population.

President Donald Trump rescinded the Obama order to close Guantanamo, but largely ignored the place. He pledged during his first campaign to “load it up with some bad dudes” but never sent anyone there and said the annual cost of operating the detention center was “crazy,” at around $13 million per prisoner.

Of the remaining prisoners, 10 face trial by military commission in proceedings that have bogged down for years. They include Khalid Shaikh Mohammad, the self-proclaimed mastermind of the Sept. 11, 2001, attacks. Two others still at Guantanamo have been sentenced and one of them, former Maryland resident Majid Khan, is expected to complete his sentence next month.

The other 27 include 13 who have been cleared for release, including eight under Biden who could now be returned to their homeland or resettled elsewhere. Two dozen have not been cleared and have never been charged, and likely never will be, a status that some Republicans continue to defend, including in a Senate hearing last month.

“We’re not fighting a crime. We’re fighting a war. I don’t want to torture anybody. I want to give them due process consistent with being at war, and, if necessary, I want to hold them as long as it takes to keep us safe or we believe that they’re no longer a threat,” said Sen. Lindsey Graham, R-S.C.

A senior Biden administration official, speaking on condition of anonymity to discuss internal policy, said the National Security Council is “actively” working with the Defense, State and Justice departments and other agencies to reduce the population within restrictions imposed by Congress. The restrictions include a ban on returning prisoners to certain countries, including Yemen and Somalia, or sending any to the U.S., even for further imprisonment.

The official said the administration is committed to closing the detention center, an effort it “jump-started” after four years of inaction under Trump.

One sign of progress is the eight approved for release through a review process created under Obama. Under Trump, just one detainee was cleared and the only release was a Saudi sent back to his homeland as part of an earlier military commission plea deal.

Critics want the Biden administration to get busy repatriating or resettling the detainees who have been cleared and to restore a State Department unit devoted to the effort that was eliminated under Trump.

“Until I see some visible signs that the administration is going to do something about it, I am not heartened,” said Lehnert, the retired Marine Corps general. “If there is somebody in charge of closing Guantanamo, I have not talked to anybody that knows who they are.”

Advocates argue the administration could resolve the fate of the rest through plea agreements with those charged in the military commission cases and releasing the rest.

Biden’s low-key approach could be a smart strategy considering the political opposition encountered by Obama, argues Ramzi Kassem, a law professor at the City University of New York who with his students has represented 14 Guantanamo prisoners since 2005.

“President Biden appears to have learned from Obama’s missteps, transferring one prisoner and clearing many without being too loud about it and painting a target on his own back,” Kassem said. “Still, the administration must up the pace because, at the rate of one prisoner a year, it won’t come close to shuttering the prison.”