Developing PE file packer step-by-step. Step 1

Since I completed portable executable C++ library development, it would be totally wrong not to use it in any more or less serious project. Thus I am going to develop a packer with step-by-step explanations of what I am doing, and C++ library will make our life easier. So, where do we start the development? Maybe, from choosing some free simple compression algorithm. After short search I found such one: LZO. It supports lots of compression modes, and LZO1Z999 is the most effective by compression ratio of all available. Of course, it is not like ZIP, but its performance is close: 550 Kb file was compressed to 174 Kb with zip with maximum compression level, at the same time LZO compressed this file to 185 Kb. However, LZO has much more fast unpacker. It is also base-independent, that means, it can be placed at any virtual address and it will work without any address corrections. This algorithm will be right for us.

I will start the development from the simplest packer and unpacker, gradually making them more complicated. At first, let’s write a program, which loads PE file using my library. We will make a packer for x86 files, i.e. we will not touch PE+. So, firstly you need to download and compile my Portable Executable library in Visual Studio 2008 or 2010 (make sure you downloaded 0.1.11 version, this step is not yet compatible with 1.0 or later). After that you should create a new project. I called it simple_pe_packer and put it into the folder with the library.


Project compilation settings should match library compilations settings, otherwise it will fail to link:


We will set Multi-threaded Debug (/MTd) for debug configuration. It is required to add LZO library project to solution now, to have something to pack data. I downloaded lzo-2.06 library from author’s homepage and unpacked it to the folder with the same name inside my PE library directory (see the first screenshot), after that I added lzo-2.06 project to simple_pe_packer solution, also I added all *.c and *.h files from lzo-2.06 folder. Don’t forget to set compilation settings, as shown in the second screenshot. Let’s add lzo-2.06 project dependency to simple_pe_packer (right click – Project Dependencies, if you use English version, of course). Then, it is required to add include directory to build lzo-2.06:


This folder should be added to Release and Debug configurations, of course. Return to simple_pe_packer project now. Here we also add an include directory:


It points to the place where my PE library header files are stored. If you placed everything like I did, your paths will be the same as mine. If not, then use your own paths.

Now turn to simple_pe_packer project. We add a new main.cpp file to source code files, this file will contain our packer code. In the beginning its code will be as follows:

This is a program, which does nothing. However, it should be compiled to make sure that all paths are set correctly. If compilation was successful, we go further. I will show main function code I’m working on or its parts only. Let’s do one small action – open PE-file:

You have to compile this code and run it for testing. Let’s run exe file of future packer using console, and give it its own name for testing:


As we see, passed PE file was successfully opened and read. In the next step we will turn to most simple packing and use MASM (or C, I have not decided yet) packer stub. And now let’s go further. At the beginning of the article I mentioned that LZO1Z999 unpacking algorithm is base-independent. It also occupies 613 bytes only. How can we compile this algorithm? Let’s create a new project configuration and call it ReleaseDecompressor – it will be designed to build unpacker procedure only. This is performed with Configuration Manager menu, choose New… in the left menu, then enter the name, choose Copy settings from: Release and check Create new project configurations option:


Now turn to lzo-2.06 project properties. In Configuration Properties – General tab change executable file Configuration type to Application (.exe). Then select all .c files in the project, except lzo1z_d1.c (because it contains required unpacker implementation), and open their properties. Exclude them from build:


We should see something like this:


Now go to lzo1z_d1.c file settings – the file, that wasn’t excluded from build. On C/C++ – Optimization tab select optimization by size (Optimization – Minimize size (/O1)), then choose Favor Size Or Speed – Favor Small code (/Os). Now get back to lzo-2.06 project settings, go to C/C++ – Code Generation tab. Turn off C++ exceptions (Enable C++ Exceptions – No), turn off buffer check (Buffer Security Check – No (/GS-)). Then, turn off manifest generation on Linker – Manifest File tab (Generate Manifest – No (/MANIFEST:NO)). Let’s turn off debug information generation (Generate Debug Info – No) on the Linker – Debugging tab. We can set Windows subsystem (SubSystem – Windows) on Linker – System tab, but that doesn’t count much. On Linker – Advanced tab set the entry point (Entry Point – lzo1z_decompress), to make resulting binary file free of CRT code. That is all, now we can build lzo-2.06 project. As a result we get small (1.5 kb) .exe file. If we open it in any PE viewer, for example, in CFF explorer, we’ll see that it doesn’t have any directories, and this is good. No imports, no relocations (we did not turn them off though) – algorithm is totally base independent! You can see, that the only code section has virtual size 0x265 (or 613 bytes):


I am sure, that if you dig deeper into build settings, you can reduce packer size for another hundred of bytes. Resulting unpacker binary code will be used in our PE files unpacking algorithm later.

That’s all, see you at the next step!

For those who are interested I share complete project with all presets and required files (however, you have to download and build PE library on your own, and set up library project as described in the beginning of this step): own-packer-step1

1 thought on “Developing PE file packer step-by-step. Step 1”

Leave a Reply

Your email address will not be published. Required fields are marked *