House Owner is charging a lot from the security deposit

I had been living in a rented accommodation, a 3 bhk individual house since 4 years. House owner was not keen on renewing the agreement after the 1st year so I did not do so. At the end of 4th year I was asked to vacate. During the notice period of 3 months we agreed that 2 months of rent would be deducted from security deposit. He wanted one month rent for painting charges. After vacating, he took the keys back after inspecting the house stating he would return the amount deducting on the whole 3 months of rent from the deposit, which includes the painting charges. He did keep all the conversations oral and never committed anything in written. I unfortunately trusted him. But now, he has been showing exponential charges and has returned only 40k of the 1L deposit that too after a lot of pleading from my end. He was not ready to give even that. I feel I am begging for my own money. He also keeps shouting when I speak to him over phone. He is not ready to give / show any bills of the repairs he says has undergone as I am sure I maintained the house well and he confirmed it while I vacated. I even cleaned the house maybe not spot clean. Not sure what to do. Kindly advise.