The Military Rabbinate was formed in the days of the Haganah, as "a department for religious manners", and was responsible for the "religious service" in the IDF. The Rabbinate was established due to pressure by the National Religious Party of the Hagganah who raised concern over problems in various military bases with the keeping of Shabbat and Kosher.

A conference was held in February of 1948, overseen by the Head of the Haganah command, Yisrael Galili, the then Chief of General Staff, Yaakov Dori, and delegates of the National Religious Party. During the conference, Galili announced the formation of a special division in the army that would attend to religious matters. The new division would be commanded by Lieutenant Colonel Natan Gordo.

The duty of the division was to ensure Shabbat would remain a day of rest, and oversee the work needed to be done in preparation of Shabbat, the supervision of Kashrut dietary laws, the nurturing of Judaic values in military bases, spiritual education, cooperation with Religious institutions and maintaining a location for housing and training for yeshiva students in the military system. The new division was named, "The Religious Service", and its apparatus and its military ranking was then established. Correspondingly, the Military Rabbinate was established, commanded by Major General, Rabbi Moshe Goran.

The confrontation between the Religious service and the Military Rabbinate was unavoidable, thus requiring a signing of an agreement between Major General Goran and Lieutenant Colonel Gordo establishing the duties and joint cooperation between the two divisions. Shortly after the signing of the agreement, it was decided that there was no need for two separate religious institutions and so the two were merged into one unit. Rabbi Gordo left his position and in December 1948, the Chief of General Staff Dori published the first General Staff command for the establishment of the Military Rabbinate, which would control all religious matters in the IDF.