At least two children are reported to have died in the explosion, which happened just before dawn in the town of Taiz.
Some reports said a basement near the building was used to store fireworks, but others said the basement held an arms dealer's cache of weapons.
Rebels in Yemen's southern region are campaigning to secede from the country.
Yemen's Defence Ministry said fireworks were stored in the basement, Agence France Presse reported.
The explosion caused the neighbouring six-storey building to collapse and damaged a third.
Earlier, medical sources told reporters that the basement was used by an arms dealer to store dynamite.