When the original 8086 was released there was only a single way of addressing memory locations. This technique uses a segment register and an offset register. These two registers are combined to compute the linear address. For example, instructions are fetched from memory by combinging the CS register (code segment) and the IP register (instruction pointer). The CS register addresses memory in 16 byte units (sometimes called paragraphs). Since the CS register is a 16 bit register, and the IP register is a 16 bit register, this method of addressing allows the 8086 to address up to 1 MB. This works because the linear address is computed by combining these two values:
CS << 4 + IP = Linear Address Of Instruction
This same technique works for the other segment registers and offset registers. This mode of addressing is known as real mode addressing.
As computers have evolved, 1MB of addressable memory has not been enough, so subsequent processors in the IA-32 family have changed to allow larger address spaces. In order to maintain backwards compatibility, the processor must be changed to take advantage of those new features. To maintain backwards compatibility, the processor starts operating in real mode (using real mode addressing) but the processor state may be changed to a different mode (normally protected mode) using whatever the appropriate mechanism. For example, protected mode addressing is enabled by setting a bit in the CR0 control register. Paging is enabled with a different bit in the CR0 control register (paging is only available in protected mode). System Management Mode Addressing is only available when running in system management mode, and that in turn is enabled by asserting a signal on the appropriate pin for the IA-32 CPU (assuming your version supports System Management Mode, of course).
To answer the original question, then. Real Mode is the execution mode of the IA-32 CPU when real mode addressing is being used. While older versions of Windows had some support for real mode, modern Windows (on IA-32) is very much based on protected mode (with paging). None of the OS runs in real mode, nor does it have any support for executing any part of the OS in real mode.
Indeed, the way that modern Windows makes older Real Mode programs work is by using "virtual 8086" machines that provide a separate execution environment - a virtual CPU. That virtual CPU can then run in "real mode" but Windows itself continues to run in protected mode.