I'm not sure there is a problem, my outdoor cat had the freedom/choice to go out or stay in, he was in a routine and would choose to go out as I went out to work and ended up waiting for me to come home. He tended to stay in on rainy days. I could put my washing out with confidence if he went out, he only let me down a couple of times in 14 years.

I didn't have a cat haven shed, or cat flap. He was fed well, had regular vet checks etc., etc..

However in this case there could well be a problem.