Payload executables generated by msfencode are commonly detected by antivirus engines, depending which antivirus engine is used. A common misconception is that the antivirus engines are actually detecting the shellcode, and therefore, the best way to avoid antivirus detection is to pick an encoder that the antivirus engine cannot handle, or encode many times. After all, those are both prominent options of msfencode.
If you just want to evade McAfee/Symantec, the basic msfencode command
cat payload | msfencode -t exe > my.exe should work fine. But if you have ever tried to evade an antivirus like Avast! that tries to catch Metasploit payloads by changing encoder or number of times to encode, chances are, you were very frustrated and just wasted time. So how can they catch every encoder and more importantly, how can you evade them?
First, let's see how Metasploit generates EXE's. The relevant code is in lib/msf/util/exe.rb in the self.to_win32pe function. First the payload is placed within a "win32_rwx_exec" block:
# Copy the code to a new RWX segment to allow for self-modifying encoders
payload = win32_rwx_exec(code)
The function is defined later in the file:
# This wrapper is responsible for allocating RWX memory, copying the
# target code there, setting an exception handler that calls ExitProcess
# and finally executing the code.
This function writes a set of assembly instructions consisting mostly of the block_api code that forms the majority of the Metasploit win32 shellcode, while randomly inserting nop instructions and opcodes to jmp over randomly generated bytes. These assembly instructions will look up and call the VirtualAlloc function to allocate RWX memory, copy the target code there and execute the code.
Ignoring the inject block for now (the same principles apply; less random code and code is just in a new section rather than replacing old code) the function then finds the executable (.text) section. Then it lists the addresses that will be modified by the loader into an array called mines:
# We need to make sure our injected code doesn't conflict with the
# the data directories stored in .text (import, export, etc)
mines = 
pe.hdr.opt['DataDirectory'].each do |dir|
next if dir.v['Size'] == 0
next if not text.contains_rva?( dir.v['VirtualAddress'] )
mines << [ pe.rva_to_file_offset(dir.v['VirtualAddress']) - off_beg, dir.v['Size'] ] end
It then finds, out of the remaining blocks, an executable block of sufficient size to store the payload with room to spare. Then it creates a sled of nops, and somewhat randomizes the entry point to land in them. At the end of the nops is a relative jump to the payload:
# Pad the entry point with random nops
entry = generate_nops(framework, [ARCH_X86], rand(200)+51)
# Relative jump from the end of the nops to the payload
entry += "\xe9" + [poff - (eidx + entry.length + 5)].pack('V')
Then it randomly changes 25% of the remaining executable bytes and the timestamp, and fixes the checksum.
So in summary, the exe's entry point is a random sequence of certain opcodes in the nop sled which does not look like a standard exe entry routine, followed by a jump to the win32_rwx_exec generated block of code that is identical to a metasploit shellcode block with some random nop and simple jmp instructions scattered inside. Then, if the antivirus engine can follow the jump to a dynamically, externally generated address, it will see the encoded payload, which will likely be much harder to distinguish than the previous block of code. In other words, if the AV is going to detect the exe as malicious, it is much easier to catch the code before the encoded payload than the encoded payload itself.
To demonstrate this, I used msfencode to generate an exe without any payload at all:
echo -n | msfencode -e generic/none -t exe > myn.exe
[*] generic/none succeeded with size 0 (iteration=1)
Despite the fact that there was no payload, upon uploading the VirusTotal, still 20 out of 41 antivirus engines, or 48.8% identified the file as malicious. See the report for yourself here.
To avoid this detection, just write your own exe or modify the source of an existing exe to get RWX memory, copy the payload into it, and execute it in a new way. Of course metasploit could do this, but then the antivirus companies, in their quest to enumerate badness, would identify whatever way metasploit used, and repeat ad nauseam. So have a little creativity yourself.
Oh, and by the way, msfencode is not useless, and neither are these encoders. They are both valuable for manually creating payloads to avoid certain bad characters, or evade an IDS in an exploit. These options just are not useful when creating an exe.