(AP) — The Secret Service prostitution scandal escalated Tuesday with the disclosure that at least 20 women had been in hotel rooms with U.S. agents and military personnel just before President Barack Obama arrived for a summit with Latin American leaders. The head of the Secret Service said he had referred the matter to an independent government investigator.

Secret Service Director Mark Sullivan, shuttling between briefings for lawmakers on Capitol Hill, was peppered with questions about whether the women had access to sensitive information that could have jeopardised Obama’s security.

Sullivan said the 11 Secret Service agents and 10 military personnel under investigation were telling different stories about who the women were. Sullivan has dispatched more investigators to Columbia to interview the women, said Rep. Peter King, R-N.Y., chairman of the House Homeland Security Committee.

“Some are admitting (the women) were prostitutes, others are saying they’re not, they’re just women they met at the hotel bar,” King said in a telephone interview. Sullivan said none of the women, who had to surrender their IDs at the hotel, were minors. “But prostitutes or not, to be bringing a foreign national back into a secure zone is a problem.”

King said it appeared the agency actually had “really lucked out.” If the women were working for a terrorist organisation or other anti-American group, King said, they could have had access to information about the president’s whereabouts or security protocols while in the agents’ rooms.

“This could have been disastrous,” King said.

The burgeoning scandal has been a growing election-year embarrassment for Obama, who has said he would be angry if the allegations proved to be true.

At the White House, Obama was asked at the end of a Rose Garden event whether he believed Sullivan should resign. The president ignored the shouted inquiries; his spokesman later Obama had confidence in the Secret Service chief.

“Director Sullivan acted quickly in response to this incident and is overseeing an investigation, as we speak, into the matter,” White House press secretary Jay Carney said.

On Thursday, eleven Secret Service agents were recalled to the U.S. from Colombia and placed on administrative leave after a night of partying that allegedly ended with at least some bringing prostitutes back to their hotel. On Monday, the agency announced that it also had revoked the agents’ security clearances.

At least 10 U.S. military personnel staying at the same hotel were also being investigated for their role in the alleged misconduct.

Two U.S. military officials said they include five Army Green Berets. Both officials spoke on condition of anonymity about an investigation that is still under way.

One of the officials said the group also includes two Navy Explosive Ordinance Disposal technicians, two Marine dog handler and an Air Force airman. The Special Forces Green Berets were working with Columbia’s counterterrorist teams, the official said.

The agents and service members were in Colombia setting up security ahead of Obama’s three-day trip to the port city of Cartagena for a summit attended by about 30 other world leaders.

People briefed on the incident said the agents brought women back to Cartagena Hotel Caribe, where other members of the U.S. delegation and the White House corps also were staying. Anyone visiting the hotel overnight was required to leave identification at the front desk and leave the hotel by 7 a.m. When a woman failed to do so, by this account, it raised questions among hotel staff and police, who investigated. They found the woman with the agent in a hotel room and a dispute arose over whether the agent should have paid her.

While the identities of those being investigated have not been revealed, Maryland Republican Senate candidate Daniel Bongino told The Associated Press Tuesday that his brother, an agent who was on duty in Colombia, is “cooperating” with the investigation. Bongino, a former agent himself, insisted that his brother was not a target of the investigation.

The Secret Service has insisted that Obama’s security was not undermined by the incident, which happened before he arrived in Colombia.

Sen. Susan Collins of Maine, the ranking Republican on the Homeland Security Committee said Tuesday that “20 or 21 women foreign nationals” were brought to the hotel. Eleven of the Americans involved were Secret Service, she said and “allegedly Marines were involved with the rest.”

In at least one of his briefings with lawmakers, Sullivan said he was calling on an inspector general to hold an independent review. Sen. Chuck Grassley, R-Iowa, welcomed that news, saying an independent review “should help the agency regain some respect from the American taxpayers and from people around the world.”

The Secret Service did not immediately respond to a request for comment about Grassley’s account.

Meanwhile, a person familiar with the agency’s operations said it was unlikely the agents involved would have had access to detailed presidential travel itineraries or security plans. Those materials are often given to agents only on the day they carry out their assignments and are kept in secure locations, not hotel rooms, the person said, speaking on the condition of anonymity.

Danny Spriggs, a 28-year veteran of the service and a former deputy director, said there was no doubt that the agents had put themselves in a compromising situation in which security could have been affected. But he said the incident did not reflect a systemic problem.

“I think we need to be careful not to paint that incident and paint the agency with a broad brush,” said Spriggs, now the vice president of global security for the AP. “The vast majority of the men and women of the Secret Service conduct their duties with the utmost professionalism.”