As we know, Wano was ruled by Kozuki Clan until Kozuki Sukiyaki was replaced with Kurozumi Orochi who in all appearances, lived extravagantly and used Kaido as shield. New generations were taught to hate Kozuki clan.
Similar History: Ancient Kingdom was ruled by an individual until the 20 Kingdoms replaced it with the World Government. The descendants of nineteen of the 20 Kingdoms were later known as Celestial Dragons who lived extravagantly and used World Government as shield.
New generations know nothing about Ancient Kingdom.
Oda-sensei has shown us that people in Wano comes in various shapes and sizes.
Kiku (who is taller than Luffy and Zoro) is smaller than Urashima.
This could imply giant heritage, similar with the people of Fishmen and Merpeople.
Why ran to Wano?
Because of Seastone. It was also proposed that Devil Fruits were an invention of Ancient Kingdom; hence, who else knew its weakness better than the inventor himself?
Wano and Isolationsm
After escaping to Wano, the isolationsm began. Why? Opening their home led to the destruction of Ancient Kingdom, so lesson learnt.
Apparently, people of Wano already knew about Haki…
…which can be imbued into inanimate objects, e.g Kuja’s Arrows
So, Poneglyphs were made on strong blocks which were made indestructible by Haki.
In short, Wano people were loyalists of Ancient Kingdom who practiced Isolationsm to protect themselves until Orochi brought down Kozuki Clan with Kaido’s help.
*Theory by Lily