Please please kindly help us asap

I have agreed to sell my house to a person. we have made an agreement on 100 rs bond paper, and the buyer kept it with him. the buyer has given some amount as advance. but the buyer continuously failed to pay the money on the specific dates which were mentioned in the agreement. but we waited patiently so that he will pay the money. and when we asked him why he wasn't paying, and when he would pay, he told us that he is not able to arrange the money and thus he cannot pay any time sooner. but we were drastically in need of money so requested him to try. but he said that he cannot arrange the funds right now and so we cancelled the agreement (buyer initiated and agreed to cancel the agreement). the buyer said that he will give us the agreement paper once we return the advance he gave. since we had no money then we told him that we will return as soon as we sell our house to some other person. and the buyer, being an close acquaintance of us, agreed for it. the buyer also bought few of his friends who were interested in buying our property. at last we got another buyer and made an deal with him, and even returned the advance amount of the original buyer and the agreement made with him was cancelled (he tore the paper right in front of us and and the new buyer) NOW, he is threatening us that he has a xerox copy of the agreement and that he will put a case on us because he has got a severe loss because of us. He is demanding extra money as he says that the property was sold without him being informed and that we made a deal with both the buyers at the same time. we are in total confusion now, as the one who actually is about to buy our house is thinking not to do it, as he is scared to be be involved in a court case. we do NOT have any written proof for saying that the agreement was cancelled or for showing that the amount paid by buyer was returned to him completely. can somebody please help us as soon as possible?