Boos' former species

The species Boos were in life, as well as similar species such as Peepas and Bonneters, is up for debate.

Spirits such as Ghost Toads, Eeries, Portrait Ghosts, and Octoboos bear similarities to other known species, suggesting that these are their undead forms. Boos have no such parallels, despite being the most commonly encountered ghosts.

Bob-ombs
Boos and Bob-ombs share similarities in body and eye shape, with Bomb Boos suggesting a further link between the two species.

However, in the promotional video "Mario Myths with Mr Miyamoto", series creator Shigeru Miyamoto denied this theory.

Shy Guys
TV Tropes' [//tvtropes.org/pmwiki/pmwiki.php/WMG/SuperMarioBros Wild Mass Guessing page] for the series posits that Shy Guys become Boo Guys in the afterlife, but their true forms are Boos, which could be revealed by removing their masks.

Other ghosts
From the same TV Tropes page, it is proposed that Boos are the result of other ghosts "dying" and being unable to reassume the form of their original species. It also suggests that King Boo provides the template for this Boo form, though his origin in turn is not discussed.

Nothing
It is possible that Boos are ghostborn and have always lived as Boos. If this is the case, it could be intended to prevent the series' audience from thinking about death and to avoid taking a stance on the afterlife, at least in the context of the main-series platformers.

The existence of towns of Boos, and of Bonneton, if its residents are considered a related species, may imply that Boos are naturally occurring. However, these locations could also be set aside for spirits to inhabit.

This theory is contradicted by other material, however. For instance, in Paper Mario, Goombario says this about Igor, a Boo shopkeeper:

"Maybe he was a merchant before he became a Boo."

While Goombario could be misinformed, this line of dialogue points toward Boos being the ghosts of a different species.