Backout of Real Estate Purchas agreement

Hello Sir/Ma'am, My name is Luke Burri residing in USA and I have made an apartment purchase agreement in Jan, 2015 in Chennai and the builder has orally promised to hand over the property by March and later said by May they would handover. In believing this I have come to India on Jun4th to do housewarming things etc. However, the builder doesn't seem to have any plans to handover the property anytime soon. The property cost is 47lacks and I have paid 5lacks at the time initial agreement and paid 35.16lacks a month ago thru SBI home loan. So, with all this hassle, I am having second thoughts to proceed with the purchase and would like to cancel the agreement and called the builder and to my surprise, the builder's response was very rude in manner and talked to me very disrespectfully and said, I may have to pay 10% as a penalty for breaching the agreement. I am really disappointed, frustrated and seeking your help to know my rights as a consumer. Please advise. Your quick response is much appreciated. I am willing to proceed if this process requires to go with a lawyer.