Winston Peters has been kicked out of Parliament's debating chamber for interjecting "that's crap" during Question Time.

Finance Minister Bill English was speaking at the time but Mr Peters had just had a run-in with Speaker David Carter.

Mr Carter thought the remark had been about his own ruling, which Mr Peters denied.

"I was talking about the minister's answer," he said.

"I took it very much to be a criticism of myself," Mr Carter said, ordering Mr Peters to apologise or leave.

"I don't want to waste parliament's time by apologising to you over this matter," Mr Peters said as he left.

"It was crap and it is crap."

Labour went in to bat for Mr Peters after he was ejected, but had no luck getting the ruling changed.