I was exactly the same. I fractured my skull 16years ago in a riding accident and didn't ride again until 18months ago. I went riding with my sister and I felt such a fool and a woosy. My sister forgot that I hadnt ridden for 15years! I have started having lessons and hacks with a local riding school and my confidence started to return. It didn't take me long to get the rhythm back for canter and I was surprised when my instructor told me I have a good seat. Although I'm still apprehensive cantering when hacking out, I am definitely not stressed about it now. I went on a 3 hour charity beach ride on a horse I hired that I'd never met before on 11th April and it was amazing.
So my advise is to get some lessons at a good local riding school and mix lessons with hacks at your own pace. And remember to have fun