The short answer is that inaugurations are held in the Capitol because Congress wants them to be held there.

Some of the inauguration practices are due to precedent, but the First Congress was quite concerned about proper procedure for the first inauguration of George Washington in 1789. Congress had come into session on 4 March 1789, but it took until April 1st in the House, and April 6th in the Senate, to have a quorum for business. Congress appointed a joint committee to plan “the ceremonial of the formal reception of the President of the United States,” which deliberated for several days before coming up with the inauguration procedures. This joint committee was charged with considering “the time, place, and matter, in which, and of the person by whom, the oath prescribed by the Constitution shall be administered to the President of the United States.” 

The joint committee initially decided that the President be “formally received by both Houses in the Senate Chamber,” and that then they all go to the House chamber, it being larger, for the oath of office, “in the presence of both Houses,” with the oath “administered by the chancellor of the State of New York,” the highest legal officer in that state. However, after further deliberation, it was decided by the joint committee that the oath should be administered “in the outer gallery adjoining the Senate chamber.” This change was passed by both houses; the outer gallery was outside and thus open to public view. 

On 30 April 1789, the House, led by the Speaker, filed into the Senate chamber. The joint committee escorted George Washington into the Senate chamber, and the chairman of the joint committee introduced him to the Senate and House. The Vice President (who had taken the oath of office a number of days before), being president of the Senate, conducted Washington to his chair, and announced that “the Senate and House of Representatives were ready to attend him to take the oath required by the Constitution.” The President, Senate and House all went out to the balcony and Washington took the oath. After he was announced, Washington proceeded to deliver his inaugural address. 

Congress has usually followed this procedure in subsequent inaugurations. For 2009, S. Con Res 67, passed in February 2008, established “the Joint Congressional Committee on Inaugural Ceremonies.” Composed of three Senators and three Representatives, it is authorized to make the necessary arrangements for the inauguration.