In 1846, the U.S. Patent Office was on the north side of F Street between 7th and 9th streets. On display on the first and second floors were patent models. In addition to patent models, the central Grand Exhibit Hall contained expedition artifacts, animal, vegetable, and mineral specimens, portraits by Charles Bird King, and the Declaration of Independence and other documents. These exhibits were called the National Museum. Adjoining the structure was a 148 square foot greenhouse containing botanical samples returned from western expeditions.
This spot in L'Enfant's city plan was marked for a national church or a pantheon for national heroes. The old Patent office building on this site was destroyed by fire in 1836. Later that same year, William Parker Elliot's design for a replacement was approved. Robert Mills became the supervising architect creditted with fire-proofing the new structure. The building was finally completed in 1867.
During the Civil War, the building that Whitman called "the noblest of Washington buildings" served as barracks, and hospital, and held Lincoln's second inaugural ball in 1865.
The building currently houses the National Museum of American Art, the National Portrait Gallery, and the Archives of American Art.