In a referendum, almost 62% voted against the progressive draft.
The margin of the defeat is much larger than opinion polls had suggested.
Chile's President Gabriel Boric, who had backed the new constitution, said he would work with Congress and civil society to come up with a "new constitutional process".
"We have to listen to the voice of the people" who, he said, had clearly not been satisfied with the proposal put forward by the constitutional convention.
He said he would continue working to reach a proposal that would "fill us with confidence and unite us all".
The process to replace Chile's military rule era constitution started three years ago after mass protests rocked the nation, which is normally seen as a haven of stability in the region.
Almost 80% of Chileans voted in favour of replacing the old constitution in a referendum in October 2020.
But the new document, drafted by a constitutional convention whose members had been chosen by voters, proved too radical for many.
It would have declared Chile a "plurinational" state, recognising the rights of Chile's indigenous populations - which make up about 13% of the population - to their lands and resources.
The now-rejected draft would also have changed many of Chile's institutions, such as replacing the Senate with a Chamber of Regions.
It also included key demands by women's groups such as as the right to abortion and requiring by law that women hold at least 50% of positions in official institutions.
While opinion polls had predicted a "no" vote, the overwhelming rejection - 61.9% against compared with 38.1% in favour of the new constitution - is a slap in the face for President Boric.
The 36-year-old leader was swept into power after the mass protests and his youthful, left-wing cabinet had promised to overhaul Chile's institutions.
But almost six months into his presidency and after the resounding defeat of the constitution which he backed, he is now expected to make changes to his cabinet to bring in more moderate and politically experienced politicians.
Analysts think that the fact that voting was obligatory meant that voters who had even slight doubts about the text chose to reject it in the hope that a new version would prove more to their liking.
Monica, a voter in the capital, Santiago, told AFP news agency: "Chile needs change, but it does not need communism, and that is what this process was attempting. It was creating inequality and division in Chile."
José Burgar told AFP that he thought a better text could be achieved: "I can assure you that changes are needed, undoubtedly there are going to be changes, but then we need a good constitution, a constitution that represents us all."
Some people took to the streets of Santiago to celebrate the constitution's rejection.
President Boric said he would now work to achieve a "text that will incorporate the lessons of the process and win over a broad majority of citizens".
However, it is not yet clear what the process of redrawing the draft will look like and how long it could take to come up with a new text.