I agree that just because he looks out the window doesn't mean he wants to go out, cats love watching the world go by from a window. If he's not attempting to get out then I'd be inclined to keep him in.

I have an indoors cat but previously I always let my cats out. But now I'd never have a cat that wandered at will, I'd worry too much. My cat does go out supervised into our garden, but she's not really that bothered, she just does a perimeter fence check and goes back in

If you do decide to keep him in make sure he has lots of stimulation indoors e.g. pouncing games, she he can use his natural instincts and keep fit.