For reasons that are irrelevant, I needed to fuzz a Windows specific codebase recently and ended up neck deep into it. I had source access to the target and wanted to try out the latest and greatest LibAFL with all the good stuff it has to offer.

Since LibAFL is more of a toolbox for building various types of fuzzers, it could already be used in certain ways in Windows. However, I was hoping to use the LLVM passes for coverage, cmplog, etc.

I also got an extra boost in motivation when I saw the open ticket for adding this feature.

Update: the PR has been merged 🎉 !

Changes required to the codebase

For starters, a few changes needed to be made to the codebase of libafl_cc, one of the projects in the main rust workspace of LibAFL. This projects serves two purposes:

1. It implements a clang wrapper such that various arguments are automatically added when compiling your target. If you are used to AFLplusplus, then you are probably used to seeing afl-clang-fast. In this case, libafl_cc plays a similar role.
2. It builds the associated optimization passes, which at the time of writing this are: afl-coverage-pass.cc, autotokens-pass.cc, cmplog-routines-pass.cc and coverage-accounting-pass.cc.

To be fair, the changes to the codebase were not super big and after thinking about it, decided to rebase a bunch of small (sometimes conflicting) commits into a single one. If anything, the best part here was some refactoring to make the codebase (subjectively) more easy to grok.

The biggest change was using clang-cl instead of clang for building the optimization passes on Windows. The reason here was to remaing compatible with the existing idea of using llvm-config to identify libraries, library paths and other flags. While on Unix platforms llvm-config generates flags for the regular clang frontend, on Windows llvm-config.exe generates flags that are compatible with Microsoft's cl compiler and the clang-cl compatibility frontend.

At this point, if you are an LLVM on Windows connoisseur, you might be wondering where llvm-config.exe came from and how optimization passes even compiled. For that, I have to recompile my own LLVM, clang and while I was at it, lld.

Building LLVM

There were mainly one reason to build LLVM myself, namely compiling llvm-config.exe. Unlike the Unix distributions of LLVM, the one for Windows does not include this binary. It is not fully clear why, but it is considered unstable for regular use. To be fair, I myself noticed some oddities when using it via a shared folder and seeing different outputs in VMs or docker containers versus the host. In any case, by simply building LLVM from source, you get llvm-config.exe for free.

Another thing I encountered were mentions of  an LLVM_EXPORT_SYMBOLS_FOR_PLUGINS variable in cmake. I found this bug report which mentions symbols not being exported unless explicitly requested. Apparently these symbols are required for compiling and using plugins on Windows. I tested with and without setting this variable, and things seemed to work just fine in both cases. I am mentioning it here for completeness.

Lastly, I bumped into an issue with the default link.exe linker. The solution was to compile lld as well and use that instead.

For reference, the powershell commands I used to build my version of LLVM were

mkdir build; cd build

cmake -G "Visual Studio 17 2022" -A x64             `
    -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld"  `
    -DLLVM_EXPORT_SYMBOLS_FOR_PLUGINS=ON            `
    -DLLVM_TARGETS_TO_BUILD=X86 -Thost=x64          `
    ../llvm

cmake --build . --config Release

If you are following along, remember to also add the bin directory to your path

$env:PATH = "<path to git repo>\build\Release\bin" + ";" + $env:PATH

LLVM runtime library

As a detour, I wanted to write down my notes regarding the default Windows runtime used when building LLVM. While running cmake, one might notice the specific text

-- Using Debug VC++ CRT: MDd
-- Using Release VC++ CRT: MD
-- Using MinSizeRel VC++ CRT: MD
-- Using RelWithDebInfo VC++ CRT: MD

The four options Debug, Release, MinSizeRel and RelWithDebInfo refer to the standard options for the CMAKE_BUILD_TYPE variable. In the above example, the code is compiled with the Release configuration.

The opions for the CRT however refer to the Windows runtime libraries. To (overly)simplify things, there are two main options: MT for the static library libcmt.lib and MD for the dynamic library msvcrt.lib. An extra d can be appended to use debug builds of the runtimes. By default, the MD option is picked up, however if you want to change that using four cmake variables: LLVM_USE_CRT_DEBUG, LLVM_USE_CRT_MINSIZEREL, LLVM_USE_CRT_RELEASE and LLVM_USE_CRT_RELWITHDEBINFO.

Using StdFuzzer

With all the above in line, the last step was to use LibAFL itself. I tested things out on StdFuzzer, which is a quick to spin fuzzer, similar to libfuzzer but based on LibAFL. After updating the code to work with the latest version of LibAFL, it was good to go.

The last thing needed here was to tell rustc to link the code against the static runtime (i.e. libcmt.lib) instead of the dynamic one (i.e. msvcrt.lib). This was needed due to some oddities regarding symbols defined in multiple libraries. While goolging for stuff, I found out others had fixed the issue my simply changing the runtime, which also worked for me. To do that, I set up the RUSTFLAGS environment variable as explained in the docs

$env:RUSTFLAGS='-C target-feature=+crt-static' cargo build --release

There is already a libafl_cc wrapper in StdFuzzer, which I then used to compile a silly example as

.\StdFuzzer\target\release\libafl_cc.exe `
    target.c -o target.exe               `
    -loleaut32 -lole32 -luserenv         `
    -O1 -g                               `
    '-Wl,/subsystem:console'             `
    -fuse-ld=lld

I used the '-Wl,/subsystem:console' argument to tell the linker what subsystem to generate the final .exe for. This is a link.exe flag, but lld understands it as well. An alternative would be to implement the main() function in the harness.

Also notice the -fuse-ld=lld argument, which was needed to address an issue with link.exe. Without the argument, link.exe would throw an error message saying fatal error LNK1190: invalid fixup found, type 0x000B.

Dockerizing stuff

For those who wish to test things out quickly, there is a Dockerfile available which sets up a Windows container with msvc, cargo and a few other requirements to compile LLVM and build StdFuzzer.