That affidavit may well have been a deed of transfer, the effect of signing which is to transfer the ownership of the house. He cannot be stopped legally from entering the property as he may well have become an owner after you signed the affidavit. Now the only legal recourse available to you is to immediately file a suit and get the deed cancelled.