I had only looked at that pattern, I hadn't actually printed out the pattern pieces. While I was cutting out the fabric I was just guessing. Turns out I should have printed the pattern out and it might not have taken as long as it took me to puzzle it out on my own.
I had to sew the brim three times- cause the first time it was too floppy, the second time I messed up, by the third time I think I finally fixed it. I used a piece of stiff felt inside the fabric to strengthen the brim. Little Man loves is the fact it's reversible; it's like he has two sunhats in one!