I agree.
I have had many rabbits neutered as i think this is the right thing to do for health and bonding reasons.
Do not let that vet put you off. The vet you choose should be experienced and confident and encouraging you to have your rabbit neutered.
An experienced vet would also do health checks before the operation and look at his previous records to make sure he has had no history of illness that could effect the fact that he is being knocked out for treatment.

As long as your rabbit is healthy you should not have any problems, better still if it is a buck rabbit the procedure is quicker and less of a risk than a doe who would have to be fully opened up to do the procedure, this takes much longer.