Construction official John Setka says crossbench senators who don't like his campaign against proposed anti-union laws might need to "toughen up".
The CFMMEU Victorian secretary also denies threatening crossbench senators Jacqui Lambie, Rex Patrick and Stirling Griff.
But Senator Patrick believes Mr Setka's comments go beyond normal campaigning and he has referred them to the Australian Federal Police.
"Mr Setka foreshadowed members of his organisations crossing paths with myself and Senator Griff at some future time and engaging in abuse," Senator Patrick told the upper house on Wednesday.
The Centre Alliance senator will also ask the upper house on Thursday to refer Mr Setka to parliament's powerful privileges committee over allegations he tried to intimidate them into blocking legislation.
The Senate is considering proposed coalition government legislation that would make it easier to ban rogue unions and their officials.
Mr Setka told a recent shop stewards' meeting the senators would "wear the consequences" of their decision, and they can expect to be abused on the street in 20 years time.
"Someone is going to point the finger and say, 'There's them f***ers that voted for that bill that f***ed up not just construction workers but all workers in Australia'," Mr Setka said in the meeting last week.
Mr Setka played down the comments on ABC radio on Wednesday.
"There's been no threat made," he said.
"Maybe they should toughen up a little bit because it's called campaigning."
Jail and fines could apply if the privileges committee finds a serious contempt of parliament.
"If we care about our deliberations conducted free of external threat of coercion, then a very clear signal needs to be sent that this is not a situation that will ever be tolerated," Senator Patrick said.
Senator Lambie has asked Mr Setka to step down from his union job. He says it wasn't because of his conviction for harassing his wife, but rather because the senator thinks he's become a target.
But Mr Setka says while he was in a bad place when police were pressing blackmail charges against him - which were later dropped - he and his wife received counselling and were doing well.
Opposition Leader Anthony Albanese wants Mr Setka kicked out of the Labor Party, but the union boss says he has no reason to step aside.
"This all started when I said I was not going to give any more money to the ALP," he said.
Mr Setka says his union is delivering some of the best wages in the country, workers are safer, buildings are done ahead of schedule, and there are no strikes.
"Why are we so bad?" he said.
Industrial Relations Minister Christian Porter told parliament Mr Setka's union repeatedly broke the law and did so to the detriment of projects such as new schools and hospitals.
Mr Porter pointed to a $34,000 fine against the union for faking a safety complaint on a school construction site.
"The reason why we pay 30 per cent more for the schools, for the hospitals, for the aged care, is because the CFMMEU breaks the law and they do it repetitively without any concern for the law," he said.
Australian Associated Press