Client asking for partial refund of Advance Payment

I was hired for a particular service of architectural rendering. ( i was verbally appointed for the work through a reference from my office colleague, no papers or work order was received or signed). The service consisted of providing a video of approx 13 minutes long ( made by compilations of small clips ) + still images. For this work I had sent a bill divided in 4 part payments. 1st instalment - advance payment for commencement of work which was (30%) of total amount. 2nd instalment - 30% of the total fees for providing the whole of images. 3rd instalment - 30% of total fees for draft of the video. 4th instalment - 10% of the fees. Now, I have partially worked and given all the work till the 2nd stage i.e. the set of images + a majority of short video clips. Due to some issues, technically and personally, I was unable to complete the work in the set deadline. Now the client is asking for partial refund of the advance money and threatening me to take legal action if not refunded. What should I do in this scenario, I have already completed and delivered more work than the advance payment was made for. Am I still liable to refund back part of the advance payment even a hefty portion of the work was completed and delivered on time ? (Although the complete video was not made, the images and clips till 5-6 mins of time was already worked upon and delivered. ) Please advise. Thank you