Is it when you touch a particular area or just touch him in general? Sounds like he's just not used to good touch yet-so maybe just touch his head every now and then and take it from their, time will bring trust for this lucky lad i'm sure
Let him take the lead for touch, as usually the best way to befriend a cat is to "ignore" them-and he will want to come and befriend you,this way it's all on his terms at his leisure and you will soon build up trust and understanding
Do let us know how you both get on and of course when he's ready-pics would be great