Born in Boston. As a young artist, he was very gifted, and was aware of the arts in the New England colonies, knew John Trumbull and Gilbert Stuart, and saved to fund his own study in London. With Franklin's help, he was taken into Benjamin West's studio as a free student. Admitted to the Royal Academy, he showed a real gift as a portraitist.

Brown never returned to his home in North America; he spent the rest of his life in London. He obtained the position of painter of portraits and history paintings for the Duke of York, and died in London.