The interpretation of builder does not seem to be sound. if that is so then it should have been mentioned in the agreement itself.
Since there is no such clarity in the agreement you should take advantage of this and inform the builder to wait till the completion of whole brickowrk.
If the builder does not relent you can file case beofre the consumer forum.