I think if you deem your neighbourhood quiet enough for him to be relatively safe, and as he's used to accessing the outdoors, then seeing as you've kept him in for 4 weeks he knows that home is home and if he's neutered it should be fine for you to let him out. It may be quieter round yours at night, our mittens like to go out for a prowl in the evening and I call them in frequently just so I know they haven't vanished! I don't have a cat flap, they miaow at the windows if they want letting in.