There is no provision under the law, which makes prostitution per se a criminal offence. However, seducing any person for the purpose of prostitution or running a brothel is illegal.
A client is guilty of consorting with prostitutes and can be charged if he engages in sex acts with a sex worker within 200 yards of a public place or "notified area.
Public place in context of this law includes places of public religious worship, educational institutions, hostels, hospitals etc.
A "notified area" is a place which is declared to be "prostitution-free" by the state government under the PITA.
Brothel in context of this law, is a place which has two or more sex workers (2a).
Prostitution itself is not an offence under this law, but soliciting, brothels, madams and pimps are illegal.
However in your case you have not been found to indulge in the prostitution activities in any public place or notified area.
Moreover you have not been caught in a raid.
The mere availability of your phone number with the broker is not a crime including the money transfer to him.
Prostitution itself is not an illegal act.
Hence your fears for the possible action by police afterwards are unnecessary.