4 Answers

Talk to your case worker or legal advisor regarding this as this requires intimate knowledge for a proper answer.

For the sake of the kids, get along, or at least do what you can in their best interest, not to spite each other!  You two do not have to be friends, or sleep together, but you are the parents, and the child has the right to be raised properly.  If you cannot, give up custody, so normal adults can raise a normal child for you! 

Is anything holding you back? If so, what? My wife filed for physical custody of our son, when the divorce is finalized. That can be years from now. In the meantime, I see him with my wife's consent, and it's o.k. with the court as well.

Without a custody order, whoever has physical custody of the child(ren) has custody. IF you let the other parent see the child(ren), make sure it is not under circumstances which can offer that person an opportunity to take custody away from you.  
My son went through this recently, and the custody order was in place about two days after he met with the family law attorney.  It went quickly because he was honest with the kids' mother about what the order meant, and he didn't try to trick her.  
You shouldn't have problems unless the orders are unfair and he refuses to sign them. If you are afraid he will kidnap the child, demand supervised visits. Be able to substantiate your claims. 

TO DIRECTLY ANSWER YOUR QUESTION, I'm going to say NO