Real estate broker

I was planning to sell a property and a real estate broker (with no RERA license) approached me that there is a buyer. I never know the broker earlier. They buyer payed the token amount agreed to pay 25% of the sales price in 10 days. After 10 days ( after I sent all the xerox documents) he cancelled the transaction saying he does not want to continue the transaction. All this communication was done through this broker. This broker initially asked me for 1% of the sales price as his commission if the sales gets completed and after registration done successfully. Though I didn't reach out to him, I agreed to it. Now the sales transaction got cancelled by the buyer himself. Do I have to pay the token amount back? The buyer is not asking for the money. But the broker is asking for the money. I told him that this token amount is not refundable, since the buyer cancelled the transaction. And also this broker has been calling me in odd times and harassing me with the calls making me not sleep. I am an NRI and getting sleep less nights due to his odd timing calls. Can someone advise for the next steps?