's Foreign Ministry said Gaddafi's wife Safia, his daughter Aisha and his sons Hannibal and Mohammed had entered Algeria on Monday morning.
The development threatened to create a diplomatic rift just as the rebel National Transitional Council
(NTC) worked to consolidate its position as Libya
's new government.
An NTC spokesman said the council would seek to extradite the Gaddafis.
A senior rebel officer also said Gaddafi's son Khamis, a feared military commander, had been killed in a clash outside of Tripoli. The report could not be independently confirmed.
Meanwhile rebel forces converged on Gaddafi's hometown of Sirte from east and west, intent on seizing one of his last bastions of support either by force or by negotiation.
Gaddafi's whereabouts have not been known since the rebels captured Tripoli and his 42-year-old rule collapsed a week ago after a six-month uprising backed by NATO and some Arab states.
Algeria's acceptance of Gaddafi's wife and offspring angered the rebel leadership, who want him and his entourage to face justice for years of repressive rule and who fear that he could orchestrate a new insurgency unless he is captured.
"We have promised to provide a just trial to all those criminals and therefore we consider this an act of aggression," spokesman Mahmoud Shamman told Reuters. "We are warning anybody not to shelter Gaddafi and his sons. We are going after them ... to find them and arrest them."
"We have heard that Algeria will harbour them till they go to another country. They are trying to go to another country, possibly an east European country," he said.
NTC chairman Mustafa Abdel Jalil called on the Algerian government -- which has not recognised the council as Libya's legitimate authority -- to cooperate with it and hand over any of Gaddafi's sons on its wanted list.
Asked if he knew where Gaddafi senior was, he told al- Jazeera TV: "If we knew where Gaddafi was now our revolutionaries would be on their way to capture him. We have no information that Muammar Gaddafi
is in Libya or in any other place."
Jalil appealed to NATO to keep up its air campaign in support of the rebels, saying Gaddafi was still a threat.
NATO warplanes have struck at targets near Sirte, on the Mediterranean coast, in recent days.
LAST STAND AT SIRTE?
Whether or not Gaddafi is hiding in Sirte to make a last stand, the city would be a strategic and symbolic prize for Libya's new rulers as they tighten their grip on the vast North African country.
Rebel forces were advancing towards Sirte from east and west even as negotiations continued for its surrender. Their eastern column had pushed past the village of Bin Jawad and secured the Nawfaliya junction By Monday.
Marwan Mustapha, an ambulance worker at Nawfaliya, said: "God willing, the rebels will enter the city without bloodshed and the negotiations will have succeeded. But if they have to enter by force, there will be blood."
In the desert to the south, Gaddafi loyalists were also holding out, notably in the city of Sabha.
But the death of Gaddafi's son Khamis, if confirmed, would be a serious blow to any chance of a military fight back.
Colonel Al-Mahdi Al-Haragi, chief of the rebels' Tripoli Brigade, said he had confirmation that Khamis was badly wounded in a clash near Ben Walid. He was taken to hospital but died and was buried in the area, Al-Haragi said.
Rebel military spokesman Colonel Ahmed Bani told Al Arabiya TV that the rebels also believed Gaddafi's intelligence chief Abdullah al-Senussi was killed on Saturday along with Khamis,
"We have almost certain information that Khamis Gaddafi and Abdullah al-Senussi were killed on Saturday by a unit of the national liberation army during clashes in Tarhouna (90 km southeast of Tripoli)," Bani said.
A U.S. official said Washington could not independently confirm Khamis' death but similar information was being received from "reliable sources".
Khamis has already been reported killed twice during the uprising only to re-emerge.
International Criminal Court prosecutor Luis Moreno-Ocampo said he may apply for an arrest warrant for Khamis.
Human Rights Watch said members of the Khamis Brigade, which he commanded, appeared to have carried out summary executions of prisoners whose bodies were found in a Tripoli warehouse.
A report by U.S.-based Physicians for Human Rights said in a report it had found evidence of crimes including "murder, torture, rape, forced internment and disappearance" carried out by Muammar Gaddafi's forces during their siege of Misrata.
The Hague-based ICC has already approved arrest warrants for Muammar Gaddafi, his son Saif al-Islam, and Senussi on charges of crimes against humanity.
A GOOD EID
The NTC, recognised as Libya's legitimate authority by more than 40 nations, was working to establish control in Tripoli after days of chaos and clashes with diehard Gaddafi loyalists.
The council, whose leaders plan to move to Tripoli from their Benghazi headquarters this week, is trying to impose security, restore basic services and revive the economy.
Gunfire echoed occasionally across the city but aid agencies reported medical and other services were beginning to function again.
Residents, hit by shortages of food, fuel and water, ventured out to shop amid the stench of garbage before the Eid al-Fitr festival which follows the fasting month of Ramadan.
"Thank God this Eid has a special flavour. This Eid we have freedom," said Adel Kashad, 47, an oil firm computer specialist who was at a vegetable market. "Libya has a new dawn."