Australia's second-biggest bank and the corporate regulator had agreed to the fine after Westpac conceded it had wrongly assessed the ability of borrowers to repay mortgages between December 2011 and March 2015.

The case centred on Westpac's use of a statistics-based household expenditure measure to automatically calculate a customer's ability to repay a loan - regardless of whether or not that measure was lower than what the customer had declared as their monthly expenses.

Four per cent of the 262,000 home loans automatically approved under the HEM method should have been referred to a credit officer for assessment, but weren't.

But on Tuesday, Federal Court judge Nye Perram said neither Westpac nor the Australian Securities and Investments Commission were able to explain why the bank decided to use that particular method.

"Not knowing ... makes it very difficult to assess how serious its conduct is and hence how appropriate a civil penalty of $35 million might be," he said.

"The Court is asked to say that the proposed penalty is appropriate even though it knows nothing of the respondent's motives beyond a bland agreed fact that it acted in good faith.

"In the information vacuum in which the parties have left the Court, this is not possible."

He concluded it was "unworkable to assess the reasonableness of the penalty if it is not known what is to be penalised" and refused to rubber stamp the fine.

ASIC had argued the penalty was appropriate given the relative size of home loans to other consumer loans, the sector's importance to Westpac's bottom line and the overall size of Westpac and its main competitors.

And Westpac's legal team had argued the court wasn't required to decide whether it was the appropriate penalty, but only had to be assured it was an appropriate penalty.

The matter has been stood over for a case management hearing on November 27.

Australian Associated Press