Regarding old watchman threat to our family as removed due to his bad behaviour

Removing watchman from our apartment We appointed a watchman a year back.He had been inflenced by two tesidents of our apatment n that to ladies. He started working everything for those two ladies. He always abuse n quarell with the flat owners. Everytime the above mentioned ladies support him in such cases. We conduct meeting n take the decisions, butthose two ladies go to presidents house n discuss the same issue n take decisions on their favour. The presodents wife is also inflenced bt them . Meanwhile recently the watchman drank n abused our president when he was in his own gaments shop. Hegot angry ,conducted meeting, all residents accepted to remove the matchman.watchman is not willing to go n started drama like crying , pleading etc. Requested time to gove two weeks. Now again those two ladies supporting the watchman saying that one the ladies will take responsibility if any thing happens . My question is, if that fellow do some thing in my case in which way i can proctect my family n remove that watchman. So i want that fellow to be removed .also be throws filled water bottles and drank alcohol used abusive language .he has been removed .but he is still roaming here and his wife and daughter are working for the those 2 supporting ladies .and creating nuisance with new watchman scaring him that they will again take duty with the support .and calling some owners blaming new watchman .how can I stop those people coming near or into the apartment how can I take action? Pls suggest