If I got it right, you are speaking about the scenario when some rogue software targets
specifically your product, right? The answer is very short - you cannot prevent it, because anyone who is equipped with a debugger/disassembler may potentially disassemble your product and analyse your protocols. The maximum that can be done here is complicating the potential attacker’s task via the techniques like encryption, various “security-through-obscurity” mechanisms, debugger detection,etc,effectively making an attack impractical. However, anyone who has sufficient dedication and some spare time on his hands may potentially break your protection,no matter how advanced it is.
Anton Bassov