Andres Ximenez, a Spanish storekeeper, built the Ximenez-Fatio House about 1798. The house is built of native coquina and reflects both Spanish and English design principles. The separate kitchen, with its brick oven, is the only original kitchen structure remaining in northeast Florida.

Architectural evidence revealed that the house and kitchen were remodeled in the 1820's resulting in a spacious boarding house or inn. The house was owned and maintained as a popular hotel by a succession of astute businesswomen.

In 1939, The National Society of The Colonial Dames of America in the State of Florida acquired the house from a descendant of the last of these women. The house has been restored to depict an early 19th-century inn.