Designing an instruction sequence so that it does something else if decoded with an offset

Saturday, July 5, 2014

This question is a follow-up to this question.

To set the context of this question, consider Null-free programming. This is a technique to masquerade a sequence of instructions (shellcode) as a string. In the C programming language, the byte 0 marks the end of a string, so the instruction sequence must be designed not to contain any such byte, otherwise it would be truncated by the string-manipulation function being abused.

The IA32 and x86-64 instruction sets, with their variable-length instructions of no particular alignment, allow instructions for task B to be decoded at an offset within an existing stream of instructions for doing task A. This technique has been used punctually in the early days of personal computing (1980s) in order to save space.

Has the technique of embedding code within code, starting at an offset within the first instruction, already been used, say, as one technique to fool anti-virus detection? Does it have a name? If it is useful and has already been used, what is an example? If the attacker is writing the code to start with, it is enough for task A to do nothing in an ostensibly harmless way, which may leave enough leeway to do anything that one could want as task B.