Money lending act karnataka

Sir, I am from karnataka, i have taken a hand loan of rs. 1lac in year 2013, at the rate of 3%, but after 15 days of getting the loan he came and took a writing from me with thumb impression and asked to mention interest rate as agreed without mentioning the exact interest rate as per agreed, once he got it he asked me to pay 6%, when I refused he asked to pay back the principal amount with 50000 as penalty I was in no position to repay, and he was quiet influential I agreed and started paying but after few months I was not able to pay the interest so he charged 200/- per day as penalty and asked me to pay 3000 per month as penalty, and added the interest of 6000/- I.e 6% to the principal amount and made sure that I was not in a position to pay interest , and this tally of principal amount reached rs.537650/-, in year 2016.he called me and told me to pay 4.5lac as one time settlement as he is doing a favour of providing concession and asked me to pay 4k per month till full amount is repaid till date i am paying him and under huge stress not able to even inform the police as he may call and threat my parents as they are very sensitive people, till date i have not said anything to them, so, I want to know is there any way to deal with this matter in court directly without going to police station. Till today I have paid him around rs. 1.8-2lac as interest only. Looking for help.