And Stephen Davies, who has chaired the council since early this year, also said the government should retain some social or affordable housing units in whatever building emerges to replace the Sirius.

In announcing on Sunday he would not heritage list the Sirius, NSW Environment Minister Mark Speakman said the government could lose as much as $70 million in sale proceeds if it was included on the register. The government intends to sell the building to fund social housing units elsewhere.

Mr Speakman's decision was immediately praised by Finance Minister Dominic Perrottet, who said the building site was expected to attract bids of well over $100 million.

"Frankly, the Sirius building is not at all in harmony with the harbour and heritage that surrounds it," Mr Perrottet said.

"Our city deserves better, and we now have a chance to deliver a building that genuinely complements our dazzling harbour, rather than sticking out like a sore thumb."

But numerous heritage and design experts, including Mr Davies, took an opposing view.

"My response is that it is disappointing," Mr Davies said.

The Heritage Council's decision to recommend the building for listing was based on two criteria: the integrity of the building's representation of brutalist architecture; and its history as a response to the Green Bans movement of the 1960s and 1970s.

The construction of the Sirius building helped allow working class residents to remain in the area during a period of heavy construction.

Mr Davies said that, with the government already having sold social housing dwellings in the adjacent Millers Point, "some contribution in that area would be valuable".

"We aren't saying it has to be retained as social housing," Mr Davies said. But he said a percentage of apartments in the new dwelling could be retained as social or essential housing "as a continuity of the history of social housing in Millers Point and The Rocks".

The National Trust was even more critical.

"This is the most recent in a series of decisions damaging Sydneysiders 'sense of place' in pursuit of money," National Trust president Clive Lucas said.

And the impending destruction of the Sirius, designed by architect Tao Gofers, has raised international ire.

"Sirius is a great example on how to create a functional, inventive, irregular design with affordable prefab concrete elements," said Felix Torkar, a curatorial assistant at Frankfurt's Deutsches Architekturmuseum, which was been running a #SOSBrutalism to save the buildings of that style.

"Instead of going for an entirely monolithic block of apartments, the facades protrude and recess irregularly with different heights as well. The layout also considers the mixed use with senior citizens and families," Mr Torkar said.

"We are very sorry to hear that the fight to save Sirius seems to have come to an end."

Before selling the site, the government says that it will obtain an early planning approval for a new building in the area. The development application is expected to be lodged early next year.

Millers Point: a community under the hammer

CORRECTION: An earlier version of this story wrongly attributed to Mr Davies an incorrect figure that overstated the money gained from the state government's sale of Millers Point properties. Mr Davies did not use the incorrect figure, and the correct figure for the sales is $264 million.