But when that will be is being kept under wraps.

“For operational reasons, the government does not discuss the timing of removals except to say arrangements will be made for his return as soon as possible," said a spokeswoman for the Minister for Immigration and Border Protection, Scott Morrison.

The silence has angered the family of Jordan Anderson-Smith, the seven-month-old Wagga baby who died a grisly and painful death at the hands of Hoerler in February, 2000.

Hoerler was released from Long Bay prison in Sydney about 10.30am and driven off in the back of a van.

It is believed he was taken to the Villawood detention centre.

Hoerler was charged with Jordan's murder but pleaded guilty to manslaughter part way through a trial in 2002.

Medical evidence tendered to the trial showed Jordan had been tortured - his toes had been crushed with a desk fan clamp - and beaten so brutally that he suffered a lacerated liver, bruised pancreas and multiple broken ribs.

The family of Jordan's father Cecil Smith said they were "not in the loop" about Hoerler's fate, but whatever his situation they are adamant he should still be behind bars in an Australian prison.

The State Parole Authority granted Hoerler parole last week after he served his full head sentence of 10 years and nine months and then kept in custody more than half of his parole period.

"We are extremely disappointed he is out of jail," said Jordan's uncle and family spokesman, Darcy Smith.

The Smith family now fears for children in the immigration detention centre and for children in Papua New Guinea after Hoerler is deported.

"If he is in Villawood, is he segregated from the main population, or will he be roaming around where there are children?" Mr Smith asked.

"I hope the authorities are notified in Papua New Guinea so when he is back there a few plans are in place.

"I would like it brought to the attention of authorities so they know exactly where he is and and keep an eye on him.

"Someone has to look over his shoulder whether he is in a detention centre or in Papua New Guinea."

Mr Smith said he rang Mr Morrison's office twice this week about Hoerler and sent three emails, but as of Thursday afternoon was not aware of a response.

"I wanted to voice my concerns to them ... and it would have saved us sitting around second guessing what the minister or immigration department was going to do with him," Mr Smith said.

Mr Smith said although his family was aware last week that Hoerler would be freed some time this week they had not been given any notice of Thursday's release.