How to cancel an Agreement with Builder (Notary)?

Dear Sir, Greetings for the day. We entered into a notary agreement with a private builder last year. I have advanced him an amount of 5 lakhs as a token amount and he has not initiated the construction of the house for a long time. Upon pressurizing the builder has constructed the plinth beam/base and the pillars only till ground level (without slab) and left it since last six months. The builder has been absconding and is not lifting any calls and also his office is closed. I have approached local police who informed us that this person has cheated many people but they can't lodge a complaint as it is a civil case. I am ending paying both EMI and rent as I have taken a composite loan, day to day necessities ( medicines, food etc) are starting to become challenging as a mother of a 4-year-old. I have approached the bank and I am informed that they can give construction financial assistance to some other builder if I get a NOC from the current builder who is in turn demanding 5 lakhs for providing NOC. Please help me to get rid of the builder.