There really isn't a proper hat store in Jax that Ive ever seen (no, shit like Lids doesn't count). The only place you might see some are in department stores & outlets. I'd kill for a Goorin Bros store here.
Saint Augustine has The Panama Hat Company though. I've been there & its proper.