What was the mode of payment by which your husband made this contribution, and in whose name was the cheque given. If that was by cheque given to the builder, then he surely has a right over the property and will have to be paid back. If the cheque was in your name, then he can only claim the amount from you and which you can dispute by saying that it was given to you as a gift. If cash, then there is no problem.
Keeping aforesaid in mind, if your husband does not agree to mutual consent, you may have to go for a contested divorce.