The remark triggered a storm of criticism and Keller has since apologized
German Football Association (DFB) president Fritz Keller faced calls to resign on Sunday after he sparked outrage by comparing his deputy to a Nazi judge.
Presidents of the DFB's regional associations, which run Germany's semi-professional and amateur leagues, announced after weekend crisis talks that Keller had lost a vote of confidence and has been "asked to step down from his position."
DFB general secretary Friedrich Curtius was likewise asked to vacate his role after losing a confidence vote.
The turmoil comes after Keller in a recent meeting likened DFB vice-president Rainer Koch to Roland Freisler, the infamous head of the Nazi party's court in the 1940s.
Freisler was also a participant at 1942's Wannsee Conference, where it was decided that 11 million Jews should be exterminated.
The remark triggered a storm of criticism and Keller has since apologized to Koch, acknowledging that his words were "totally inappropriate, notably towards the victims of Nazism."
Keller ruled out stepping down over the incident, however. Koch has not said that he has accepted the apology.
In a statement, the leaders of the DFB's five regional and 21 state associations called Keller's Nazi reference "completely unacceptable" and said they condemned it "in the strongest possible terms."
"The president's comments are incompatible with the principles and values of the associations," they added.
According to the statement, Keller and Curtius have asked for time to consider the resignation requests.