Owner refusing to surrender Security Deposite

Dear Sir, My situation is very critical and interesting also to read this I am actually staying in a rented house without an agreement, Initially my house was rented to my two office colleagues 4 years ago, with the security deposit of 20000, however after 2 year from taking the rented house both the room partners were left the room while another person along with me were joined the same room by paying the security deposit of 20000 to my old colleagues, on top of this last year my roommate also left the room and i had paid him 10000 as shared security deposit amount, now i am single staying in the room, Due to mutual understanding with owner we had not made the rent agreement till date, now currently when i called the owner and told him that i am vacating the room with notice period and i need my security deposit then he suddenly shocked and started asking for Coloring and old furniture maintenance charges for which i totally denied, because at the time of entering in the room owner has not asked to any of us for these extra charges and we were not in commitment with him in any way, As i had told him orally only, and had some serious arguments with him on the same issue, Furthermore, he is asking for rent of notice period month also, for which i had told him to deduct it from the SD, but he is still asking for again payment of rent for the notice period otherwise he is telling to impose penalty of two month rent on me for the single month of notice period, Hence kindly requesting you to suggest me the proper way to deal with my owner, Waiting for your valuable suggestions, Many thanks in advance,