Agreement Cancellation

Hi sir, I am from Andhra Pradesh.Recently we saw a property which we want to buy and that property costs 33,40,000/- and to buy that first we are asked to make agreement between us and the seller.So that we have agreed to pay 4,00,000/- as advance payment at the time of agreement and we payed them 4,00,000/- as we have agreed and next we have moved to get the remaining amount from bank loan so that it covers up total money of the property cost and when we approached many banks they said they cannot give full amount that we are required and we have lacked in money and decided not to buy the property and then we said to seller that we didnt get the expected amount from the bank so we don't have enough money to buy your property and want the given advanced 4,00,000/- amount back.The seller said that he has used the money that we have given as an advance at the time of agreement to buy another property and he forces us to buy the property and says he will not give money back even when we said to him that we don't have that much money and we didn't get expected amount from bank loan. The agreement is not registered and written on a stamp paper. Registration of the property is due. What should i do? What is the solution? Help me please. I hope i get the solution from people