ESC is the 8086's gateway to a coprocessor. Its opcodes are D8h to DFh, followed by an ordinary ModR/M byte. Three bits from the opcode and three from the reg field of the ModR/M byte form a 6-bit code that the 8086 ignores; it means something only to the coprocessor.

The 8086 does very little itself. If the operand is a register, it does nothing at all. If it is memory, the CPU computes the effective address (any addressing mode, segment overrides allowed) and reads from it. The 8087 watches the bus, picks up the instruction and that address, and carries out the real work: it may read a longer operand or store a result there, taking over the bus to do so.

In practice

Nobody writes ESC by hand for the 8087: the assembler turns 8087 mnemonics into ESC bytes. The code D8h + 7 with reg = 0 is code 38h, which the 8087 reads as "load a 16-bit integer":

FILD WORD PTR [BX] ; 9B DF 07: WAIT, then ESC 38h, [BX]
FADD ST, ST(1) ; 9B D8 C1: WAIT, then a register operand
FWAIT ; 9B

The 9Bh in front of each is the WAIT that MASM adds for the 8087 (see WAIT).

With no coprocessor present the ESC bytes themselves still execute as plain memory reads and compute nothing. Software checks for an 8087 before using it, or links an emulator library that does the floating-point work in software.

Later CPUs: from the 80286 a flag in the machine status word can make ESC raise an exception instead, which is how floating-point emulation and task switching of the FPU state work; from the 80486 the FPU is on the processor chip.