If you're looking for a chill spot with minimal crowds and not too expensive, you might want to check out Baja, Mexico. It’s got tons of spots that match what you’re asking for—laid-back vibe, no fancy hotels, and you can usually find a good panga and some basic accommodations with no problem
A lot of the coastline is still pretty untouched, so you'll have plenty of space to explore and spear on your own without running into big groups.You can even ask locals where the best fishing spots are, since they know the area well. Just make sure you check the local regulations for any protected areas before you go.