I guess it all depends on why you are wearing the hat.
Are you wearing it to show status as a wise elderly gentleman?
We used to do demonstrations of Ju Jitsu at various public events, one sketch we used to do was me wearing a flat cap, walking with a walking stick, complete with limp (but still wearing Gi and Black Belt)
Three people used to attack me, and I knocked the crap out of them - ending up with three of us holding the walking stick. Two were facing me, and had hold of the outsides of the horizontal walking stick, and I had hold of the middle - I dropped back and down, threw the two attackers over my head, and rolled back to my feet.
Anyway - we did it one day, and came to the end section - but we couldn't all get hold of the walking stick?????. So we improvised on the spot, and Gordon and Ben (the two remaining attackers), just had one hand each on the stick, and did a lot more leaping then usual
So we were scratching our heads and looking at the stick when the chief instructor (Ged - who was the third attacker) came up to complain that I broke the stick across his back early on in the routine
I never noticed, Ban and Gordon never noticed - but Ged obviously did!
In all the rolling and leaping about, the flat cap never came off of my head, which I found quite amazing.