Seller backing out after plot sale

A defense housing organization (lets say Army Welfare Housing Organization (AWHO)) comes out with a scheme for allotment of plots for defense personnel in Location A in 1997. During the initial allotment process itself, the original allottee sold the plot on POA basis to me and all the installments including application fees have been paid by me. Due to scarcity of land in Location A, AWHO changes the location of plots to Location B. The demarcation of plots and development of area in location B is delayed for almost 10 years since there are some litigations on this land. The seller keeps quiet during all this time. Once all the legal cases were settled by AWHO, I asked the seller to get the plot transferred in my name, the seller backed out. He said that the deal is cancelled since location of plot is changed and he sold the plot in Location A only. He offers to return the amount paid by me with interest. Does the seller have a point that deal is null and void due to change in location? When I reached out to AWHO, they say its your personal matter between seller and buyer and you should settle the same in court and plot cannot be transferred in my name until the original allotee gives in writing. P.S: I have the physical possession of plot but seller is putting pressure on association and AWHO to handle possession to him.