We just learned from the latest chapter that the old and decrepit woman who advised and helped Orochi to rise to power is from the Kurozumi clan and her name is Higurashi.
She’s the previous user of the Mane Mane no Mi.
The elderly balding man who was present at the meeting between Higurashi and Orochi is also a Kurozumi and his name is Semimaru and he is the previous user of Bari Bari no Mi.
Higurashi and Semimaru were also probably the ones who introduced Orochi to Kaido.
Now this is just a theory, but it might be that Higurashi and Semimaru were part of the Rocks Pirates in the past. This is a possibility considering the fact that one of the faces that Higurashi used when she met with Orochi looked like a younger version of Shiki.
Higurashi and Semimaru probably left Wano right after Orochi’s grandfather lost his power and his life and the Kurozumi clan lost their position and posessions in Wano and after leaving Wano Higurashi and Semimaru became pirates and eventually sailed with the Rocks Pirates crew under Captain Rocks D. Xebec.