Problem with the contractor

So we recently purchased a house and added funds for the renov/upgrade. We agreed on a quotation provided by the contractor only to find out that they cannot do it right away and will need to wait for 2mos. So that was fine since we don't have anybother choice and already cancelled the other contractor that bid to the project. We agreed they will start mid december. He sent someone to make a hole on our wall for the basement window last October 30 in prep for the construction. Comes end of november, we did a followup since we don't have any layouts yet for the basement, (Full suite for the basement) only to find out that they can't start on mid december and would probably start on january which is not yet sure. We would never know that the project is being pushed again for two weeks if we did not do a follow up. He did not offer anything nor did apologize and was trying to tell me that that is how renovations work. Very unreliable and unprofessional. So we decided to just cancel the agreement and look for a better alternative. Now he is harrasing us to pay the work that was done at the basement. That is putting a hole on my wall for the window. Question is what is our rights with regards to him asking for payment? We would pay him as soon as the project is done since funds will only be released by the lender upon completion of the project. TIA