Cancelled a property deal- now being sued by the seller

Decided to buy a office property 50 lakhs from the seller. Gave token of 2 lakhs. The seller Prepared a receipt stating the token amount and the deal amount on a normal printed paper. Seller himself insisted on preparing the Draft sales Deed to be submitted to the bank for property loan. it was cleared briefed that the next payment would start after the approval from the bank. The amount mentioned in the draft sales deed was only 35 lakhs. He was not ready to change saying that he would incurr capital gain tax. After a marathon meeting between me, him and the agent from both side, i agreed to pay additional 50,000 to him because we liked the property. All agreed. Now we requested him (including his agent) to prepare a proper MOU so that there is no confusion later on, to which he declined. He told me to pay some more amount. We felt including his agent that there is no trust in the deal as the seller is not ready for MOU and so we decided to cancel the deal. The seller got angry and within 2 hours mailed me a notice claiming 50 lakhs + 1 cr for mental harassment. The seller is a lawyer and he is threatening a common man like me in his notice to the fullest. Kindly guide me