Haka

Detecting Obfuscated Shellcodes v0.1

A shellcode is a malicious payload used to exploit vulnerabilities such as buffer overflow of format string bugs. A shellcode is made of low level instructions allowing to get a shell on the victim machine or to run specific commands. Here is an example of a /bin/sh shellcode:

char shellcode[] = "\x6A\x68\x68\x2F\x62\x61\x73\x68"
                   "\x2F\x62\x69\x6E\x89\xE3\x31\xD2"
                   "\x52\x53\x89\xE1\x6A\x0B\x58\xCD"
                   "\x80\x31\xDB\x31\xC0\x40\xCD\x80";

It is trivial for an intrusion detection system to detect this shellcode. However, if the shellcode is encoded, it is more challenging for an IDS to detected the polymorphic payload.

Even if the shellcode is encoded, some invariant byte sequences are still observable. For instance, the paper ^[polygraph] shows that the bytes in yellow boxes are present in all outputs of the well-known polymorphic shellcode engine Clet for a given input (/bin/sh shellcode). Shaded bytes are present at least in 20% of generated outputs.

So, to block all Clet's polymorphic shellcodes, we define the following security rule which checks if bytes in yellow boxes are present in tcp data and raises an alert in case of matching:

local invariant_bytes = "\xeb.*\x31.*\x20\x8b.*\x74\x07\xeb.*\xe8.*\xff\xff\xff"

haka.rule{
    hooks = { 'tcp-up' },
    eval = function (self, pkt)
        if #pkt.payload > 0 then
            -- reconstruct payload
            local payload = getpayload(pkt.payload)
            if string.find(payload, invariant_bytes) then
                haka.alert{
                    description = "polymorphic shellcode detected [Clet engine]",
                    sources = haka.alert.address(pkt.ip.src),
                    targets = haka.alert.address(pkt.ip.dst)
                }
                pkt:drop()
            end
        end
    end
}

We evaluated the detection efficiency of the following rule by generating more than 10000 Clet's outputs and by replaying the Darpa pcap capture ("safe traffic"). We blocked all the malicious payload generated by Clet engine with 0 false positive alerts.

The next release will feature a new regular expression engine allowing to match malicious payload across multiple data packets and thus will allow us to easily detect the above polymorphic content even if it is split into multiple packets.