Scholz and his running mate Klara Geywitz obtained only 45.33 percent of the vote of the party's rank and file, soundly beaten by challengers Norbert Walter-Borjans und Saskia Esken, who won with 53.06 percent.
The humiliating loss for Scholz marks another sign of discontent within the Social Democrats over their partnership with Merkel's centre-right party.
It is also a prelude to high drama in the coming week with the party poised to vote on whether to stay in the coalition as they meet for the congress next Friday.
Wounded by an election rout in 2017, the SPD had initially sought to go into opposition, but allowed itself reluctantly to be coaxed into renewing an alliance with Merkel.
Many within the party however remained wary of continuing to govern in Merkel's shadow, and the coalition has lurched from crisis to crisis.
The SPD has also been battered by a series of regional and European election setbacks this year.
The leadership race was triggered by the departure of the party's previous leader, Andrea Nahles, after the party's poor showing in European Parliament elections.