THE FIRSTBORN IN ALL ISRAEL LOSE THE PRIESTHOOD BECAUSE OF SEXUAL SIN

Midrash Rabbah - Numbers 6:2

TAKE THE SUM OF, etc (Numbers 4:22).

They [the firstborn], in fact, would have been worthy of assuming the priestly office and the duties of the Levites, if they had not sinned in connection with the incident of the Golden Calf. For originally the firstborn offered sacrifices; as it says, And he sent the young men of the children of Israel, who offered burnt-offerings, etc. (Exodus 24:5) … Who caused the firstborn to forfeit all this glory? The cause was their having exalted themselves and worshipped the Golden Calf. … Thus we find that the firstborn Israelites fell from their greatness as a result of the incident of the Calf.