A brownout is an undervoltage condition, when the AC supply drops below the nominal value by about 10% (Nominal meaning 110-120 or 220-240 in most places). So in the US a brownout might be defined as the AC voltage dropping below 99V. The Intel specification for ATX power supplies specifies that voltages between 90 and 135, and 180 and 265 should allow correct power supply operation (section 3.1), so the power supply will still run normally even when a noticeable brownout occurs.
Some people also include very brief power dropouts (under 30mS, or about 2 AC cycles) as brownouts, as incandescent bulbs will briefly, but visibly, dim during that time similar to a real undervoltage condition.
In either case, Intel defines them as undervoltage conditions, and discusses what requirements an ATX power supply has to follow under such conditions in section 3.1.3 of Intel's ATX12V Power Supply Design Guide
The power supply shall contain protection circuitry such that the application of an input voltage below the minimum specified in Section 3.1, Table 1, shall not cause damage to the power supply.
Typically power supplies have an input section composed of a bunch of interesting circuitry that, at the end of the day, provides about 308 VAC to a transformer, which then powers the regulation and conditioning circuitry. This circuitry actually forms the major basis of the regulation circuitry, and if you are using less than the full wattage of the power supply may be able to manage with significant undervoltage conditions without falling out of regulation on the output side.
When a brownout occurs, the powersupply will attempt to deliver the rated current for as long as it can (based on the incoming voltage and current) and if it cannot maintain regulation it'll deassert the Power Good signal going to the motherboard. The motherboard is responsible for deasserting the power on signal going to the supply, and if it does so in time, then the supply will drop all its output and turn off.