How buffer overflow attacks work

Buffer overflows are a favorite exploit for hackers. The vast majority of Microsoft's available patches fix unchecked buffer problems -- but what about applications developed in-house? They are just as susceptible as commercial applications to buffer overflow attack. It is therefore critical that you understand how they work and perform vulnerability testing on your home-grown applications prior to deployment.
What is a buffer overflow attack?
A buffer overflow is an exploit that takes advantage of a program that is waiting on a user's input. There are two main types of buffer overflow attacks: stack based and heap based. Heap based attacks flood the memory space reserved for a program, but the difficulty involved with performing such an attack makes them rare. Stack-based buffer overflows are by far the most common and are the type I will focus on in this tip.
In a stack-based buffer overrun, the program being exploited uses a memory object known as a stack to store user input. Normally, the stack is empty until the program requires user input. At that point, the program writes a return memory address to the stack and then the user's input is placed on top of it. When the stack is processed, the user's input gets sent to the return address specified by the program.
However, a stack does not have an infinite potential size. The programmer who develops the code must reserve a specific amount of space for the stack. If the user's input is longer than the amount of space reserved for it within the stack, then the stack will overflow. This in itself isn't a huge problem, but it becomes a huge security hole when combined with malicious input.
For example, suppose a program is waiting for a user to enter his or her name. Rather than enter the name, the hacker would enter an executable command that exceeds the stack size. The command is usually something short. In a Linux environment, for instance, the command is typically EXEC("sh"), which tells the system to open a command prompt window, known as a root shell in Linux circles.
Yet overflowing the buffer with an executable command doesn't mean that the command will be executed. The attacker must then specify a return address that points to the malicious command. The program partially crashes because the stack overflowed. It then tries to recover by going to the return address, but the return address has been changed to point to the command specified by the hacker. Of course this means that the hacker must know the address where the malicious command will reside. To get around needing the actual address, the malicious command is often padded on both sides by NOP instructions, a type of pointer. Padding on both sides is a technique used when the exact memory range is unknown. Therefore, if the address the hacker specifies falls anywhere within the padding, the malicious command will be executed.
The last part of the equation is the executable program's permissions. As you know, most modern operating systems have some sort of mechanism to control the access level of the user who's currently logged on. Executable programs typically require a higher level of permissions than the user who's currently logged on, and are therefore run either in kernel mode or with permissions inherited from a service account. When a stack overflow attack runs the command found at the new return address, the program thinks it is still running. This means that the command prompt window that has been opened is running with the same set of permissions as the application that was compromised. Generally speaking, this often means that the attacker will gain full control of the operating system.