Refund from Builder and Agreement of Loan with bank - Please Help

Hello, 1 month prior i have booked a flat with a builder giving just the token money of 21k, and rest was promised to be paid after the loan was done, but there was never any contract between the parties. Now, the builder agreed to that and loan process started and after a series of difficulties finally the loan got approved, then we had gone to the bank and signed the loan documents and sanction letter, note that this does not include any loan stamp papers, it is just the Loan agreement of the bank. Now after the loan got approved and we are done signing these documents, we had a meeting with the builder again, and there we had the some conflicts and heated argument and after the builder has told us to cancel the flat and refuse to even give back the 21k token money, after that we informed the bank that we don't need that loan anymore. Now the bank is asking us to pay some claims and money as a charge like somewhere around 15k, and builder has also refused the payment of 21k, so now what can I do, Do I need to pay this amount of 15k to bank, when I have not taken any loan from them in this case. And how can I get the money back from the builder?