Delay payment claimed by the builder.

I bought a property in Dec'2016 from a renowned builder, since I was not able to finance the whole by myself, I took a home loan, however, paid the 50% downpayment and rest as a loan. In the beginning, itself builder signed up a confirmation agreement with me where it was mentioned that there will be a payment schedule which I have to follow otherwise there will be delay charges of 18% per annum after first 30 days and 25% per annum afterward. That time I did not really think about the payment schedule much because the builder himself was managing my home loan processing. The builder had its in-house home loan department, who selected the bank by themselves, prepared all documentation and submitted to the bank for approval and so on. The bank approved the money and the loan was sanctioned. All these while, I did not realize that there were some lapses in the payment schedule and after a few months, the builder started to send me a demand letter for delay charges. I have been rejecting all the letters vide email and sometimes have used hard words as well to deny the charges, but all in vain the builder keep on sending the delay payment charges even today. I have kept all the backup letters and email and all the details of the communication during the loan processing also. The payment schedule also mentioned that the last 5% to be given to the builder on after he gives the unit for the interior. The builder sent me an email to take over the unit for interior works in February 2018 however, I did not pay him yet that as there were several patch up work and incomplete works in the unit, so i insisted him to complete them first then only I will take possession of the unit, also the OC of the building is not yet available and I did not even paid the stamp duty and registration of the unit yet. How do I create a defense case against the delay charges and does the builder have any capacity to harm me anyways if i stick to my stand and do not register the unit until the balance works are complete.