Builder refuses to give flat

I booked a Flat in 2014 paying 20% of the building cost. An agreement was made. After that I could not secure a bank loan, hence balance payment was not be given in time. In the agreement, Penalty clause is mentioned clearly for late payment. Also the builder could cancel the agreement after paying back the money paid with 5% deduction. The builder's marketing agent sent me a demand notice only once in 2014. After that nobody contacted me. I too did not contact them. The building was constructed. In 2016, I could secure bank loan; then contacted the builder's marketing agent; asked them whether they will charge penalty for late payment. They did not give any written reply but told me to make the payment. No penalty will be charged. I asked for their statement in writing but they did not give any. Meanwhile the bank loan's validity got expired and it was again re-validated for 3 months. Now when I gave them letter to fix date for registration, wherein all their dues would be cleared through bank, they are claiming penalty of huge amount and threatening me with forfeit of the paid amount. My point is - If I have not paid money in time, they could have returned my money with 5% deduction. They have utilized my money in their business and now stating (verbally) that they will not give me the flat. The Flat is in Kolkata. Please advise.