As daft as it sounds..... Maybe he has lots of owners all that take him in for 10mins each day for a cuddle and he wants to go out before he misses his next fish supper lol. (hence the food left out for him) maybe thats one of his many owners sick of trying to keep him indoors for longer than 20mins
My last cat was a very outdoorsy cat, he would always be close to home...... but always outside, Dont worry about cats in rain, they will find shelter for themselves but if they have a home..... then its wrong to expect him to shelter himself constantly so if you find them. Slap their wrists lol.
You have done the right thing showing him care and attention, and i also think you have done the right thing letting him go if hes showing signs of wanting to leave.
I wish you all the luck with your cat

And maybe if noone comes forward claiming him you can practice how it would be having a cat and maybe buy yourself one

call it...... Rent-A-Cat lol.