1. Recovery agents have no authority of law to recover an unpaid loan. They have been frowned upon by the Supreme Court on more than one occasion wherein the court made it clear that recovery agents and the bank officials who rope in these agents must be prosecuted. Since then officials of many banks have been successfully prosecuted for taking recourse to such an extra-judicial measure.
2. A court official who comes to serve the court notice does not abuse the inhabitants of the house. If the person to whom the summons is issued refuses to accept it, he records a finding to this effect and goes away. So rest assured that this guy was a recovery agent employed by the bank, not a court official.
3. You should lodge a complaint for intimidation to the nearest police station against the bank and recovery agents who are harassing you.
4. If you have not repaid the loan the bank can take out recovery proceedings in a court of law against you, but it cannot take recourse to such extra-judicial measures.
5. Since the police has also washed off its hands you should engage a lawyer and file a case for criminal intimidation in the court.
6. The bank cannot recover its money now as it ought to have recovered it within 3 years.
7. No case for defamation is made out though.