Joseph Cassey, (1789-1848) the patriarch of the family, arrived in Philadelphia from the French West Indies sometime before 1808. He prospered in the barber trade and as a perfumer, wig-maker, and money-lender. Joseph bought and sold real estate, often with sometime business partner and fellow Philadelphia African American abolitionist, Robert Purvis. Among the properties that Cassey acquired were several in the neighborhood of his home near Society Hill, part of the property at 243 Delancey Street, and a Bucks County farm he shared with Purvis which was frequented by well known political activists such as Lucretia Mott and William Lloyd Garrison.
Cassey, as a landlord in 1847, collected rents from seven families totaling 26 people living at 243 Delancey Street, later to be known as “The Cassey House.” The house was a main structure facing the street along with three smaller apartments on a rear courtyard.
Son, Alfred S. Cassey, a postal worker and political activist, lived with his family in the Cassey House until 1897. Granddaughter Matilda Inez Cassey, an accomplished classical pianist, resided there until her death in 1916. Because Matilda had no children, the house passed to her cousin, Maud Cassey Mosely, of Jersey City, New Jersey. Maud and her children sold off portions of the property, with the final piece, a small apartment passing out of family hands in 1929. Today the Cassey House is listed on the National Register of Historic Places.