Without funds to move and restore the house, the Milton Historical Society sells the house to members Dr. James B. Ayer and his wife Hannah for a dollar. The Ayer's spend over $40,000 over the next four years to move the house from the 1781 lot to 1370 Canton Avenue and have it restored by architect William Morris Hunt II. Subsequently, a bank is built on the house's former location (currently Citizens Bank).

In 1963, the Ayers bequeath the house and an acre of their estate to the historical society.