Metal Harrasment by Pathan Money Lender

I have Taken 205000 /- from a licensed Pathan 4 years back in parts. After 1.6 he said he can give in my frnds name so I included my wife and a friend but not forcefully and telling everything clearly. Now the loan was taken @8% interest and against a blank cheque & kyc (each). Since 2018 I am paying 16400/- pm in cash until corona struck and due to reduction in salary I started giving less but couldn't keep up the dates , but the payment was made every month except twice. My salary was 20k but still I kept on giving 12+ every month until in Jan 23 I lost my job ad wife got T.B suspected. In February I am unable to arrange money as waiting for FNF, but like every month he kept in abusing me , my mother, my deceased father. He even tried visiting my housing but couldn't enter due to security reason. He calls and threaten like hell and as me and my wife lives only (my mother is in her native place) we are very scared of going out of him. Although payment is every month but his abusement & threating is continuos. Against 205000/- I have paid 360000/- already @ 8%. We are very new to our current place and what shall I do to stop him . Shall I file a case or contact the police if he arrives? Please help me