PHILADELPHIA — The three-story brick building stands like a ghost of what it once was, the letters “Joe Frazier’s Gym” stenciled across the facade in washed-out letters. A former destination for aspiring fighters and children seeking refuge from the gang violence of North Philadelphia, the gym is now a discount furniture store.

A sign on the window reads “Knockout Prices.”

Now, nearly a year after Mr. Frazier’s death, momentum is building to designate his former gym a historic site. The city is also working with his estate to erect a statue honoring him.

The efforts are a rethinking of Mr. Frazier’s legacy in his adopted city, which never so much as named a street after him during his lifetime. The question for many who knew and admired Mr. Frazier: What took so long?...