At BSides Las Vegas, I just released Ambush, an open-source Host Intrusion Prevention System that I have been developing for the past few months. See my talk at http://www.youtube.com/watch?v=kzgBcSHQDAs for the full motivation, description, and demonstration.
In summary, after all of my offensive research, Ambush is my effort to arm the defense. I wrote Ambush to help change a fallacy of defensive thinking (the wall mentality) and to provide the tools for the most effective defensive strategy: to understand attacks end-to-end, create and employ detections for those attacks at as many points as possible, and catch the attacker by doing something unexpected. If you have ever complained that the attacker only needs to win once or find one vulnerability to exploit, your strategy will fail.
Part of this is a reliance on intrusion detection and prevention systems built to monitor a point or set of points such as a network connection. In response, malware authors have become adept at obfuscating traffic through these points with encryption, encoding, and obfuscation. Ambush uses a different strategy, based not on another point but on the flexibility to add almost any point dynamically. This capability employs a function call hooking system that can intercept virtually any exported function or set of functions system-wide. This is in contrast to other HIPS's, which are limited to functions chosen during design and only monitor certain actions, such as file and registry edits.
For example, given the shellcode for the Eleanore Exploit Pack below, a common payload of exploit packs, we can see five different functions called. Ambush provides the ability to configure signatures alerting or blocking on any of these functions being called from outside a mapped module, or from read/write/execute memory, or by arguments, or by process, or from lower-layer functions.*
Attackers will be unable to perform almost any action without the possibility of getting caught, as any action can get caught in just about every conceivable way. Defenders now have far greater ability to do something unexpected and catch malicious activity.
Eleanore Exploit Pack Shellcode:
By arguments - WinExec's argument matching a temporary path regular expression
By process - URLDownloadToFileA should only be used by a particular process or not by Internet Explorer
From lower-layer functions - CreateProcess rather than WinExec or if InternetOpen is called from urlmon.dll, that means URLDownloadToFileA or an equivalent was called