I don't think you are being panicky at all. Letting your kitties out exposes them to all kinds of risks and some risks you don't even consider - like the little guy getting stuck. Are there any dogs, coyotes or big birds like Eagles and Hawks where you live? I would also be worried about other people. A few months ago a friend's cat was poisoned by her neighbor for no reason other than he came into her backyard and she didn't like him. She was shocked because this woman had always been very nice to her and then she poisons her cat. Also, like you said, someone might see him and decide they want him.

I would not let him out until you get him neutered because he will stray. Un-neutered male cats want to claim their territory and of course they will stray to look for a mate.

I don't mean to make you more paranoid but studies have shown that outdoor cats have a life expectancy of 5 yrs or less as compared to indoor cats who can live up to 20 yrs.

All my cats I have now are strictly indoor because growing up there were just too many incidents. My mom was (still is) a very conscientious cat owner and kept close eye on the cats when they went out but accidents happen. Our 21 yrs old cat was killed on our front porch by two dogs who had gotten lose. One of our cats got into a fight with another and contracted Feline Leukemia eventhough he was vaccinated for it. 2 others went outside and never came back and after extensive searches, signs, etc we never found out what happened.

She trained her Siamese cat to walk on a leash and she takes him out for walks. With you guy still being a kitten you might be able to leash train him.
She too has a fenced in yard and she lets her cat out on a long leash so he still has room to explore but can't go too far and so far it has worked out great.

Her other Siamese, who has since passed away, went outside but we always went out too and supervised her for an hour or so.

I understand why you want them to be able to go outside and I think if you let your cats then you should be a little paranoid. It is a big world out there.

__________________
"The greatness of a nation and its moral progress can be judged by the way its animals are treated.'
--Ghandi