This is a topic that has greatly interested me, but from the perspective of a tester. On one side, I think the ability to reverse engineer malware is fascinating, but more to the point what I really want to be able to do is see how the tools described can actually be used to augment security testing.
Malware has become one of those topics that we often wring our hands about because we know it's a threat, we want to better comprehend it, but do we dare open ourselves up to the potential of doing something wrong and unleashing an unintended havoc on our machines or networks? Fortunately, Michael Sikorski & Andrew Honig's book "Practical Malware Analysis" helps to de-mystify this type of operation, and also make it understandable from a variety of perspectives. If you are a programmer, this will be very handy. Even if you aren't, there is a lot of good ideas and techniques in this book that you can use.
Practical Malware Analysis is structured with regular chapters describing the concepts, and each chapter ends with a series of labs. the answers to these labs take up nearly a third of the book. They consist of short answers for the specific questions as well as longer form answers that go into great detail to describe the steps and the methods used to test the files and provide analysis of what was found.
Part 1 starts out by explaining what Malware is and how developers and testers can get into the files and poke around using some basic and freely available tools. The first part of the book focuses on performing static analysis of files and looking inside them to understand what might be hiding in the files, along with ways o read the headers, strings and data hidden in the files. To get beyond the static analysis, the users are guided through setting up a virtual machine environment using VMWare. Dynamic Analysis is what happens when a piece of malware is actually run, and the determination is made that static analysis just won't cut it any longer. From here, we need to actually start poking at the malware and see what it will do in real time (yep, in that safe VM, using what's referred to as a "Malware Sandbox" consisting of a variety of free tools).
Part 2 of the book goes into greater detail regarding performing more advanced Static Analysis techniques, including dis-assembly, specifically dis-assembly on x86 Architecture processors, reverse engineering using tools such as IDA Pro (it gets its own chapter and shows up in the future labs), examining how to get a high level view of the code using code constructs, and how to use the tools to recognize when we are seeing those code constructs in assembly language. since Windows is the target of choice for most malware, Chapter 7 goes into specifics about how to recognize and work with the Windows API and use the registry editor to analyze registry code. Processes, threads, and network functionality are also primary considerations.
Part 3 brings us into the realm of more advanced Dynamic Analysis. Starting with debugging at both language and assembler levels, we see how to step through programs a piece at a time so we can see what is happening or change the way the code executes. Dedicated chapters to OllyDbg (a user level debugger) and WinDbg (a kernel level debugger) show how to do these steps with these specific tools.
Part 4 focuses on the way that malware behaves and what sets a program apart as being malware. It examines a variety of back doors in the system and ways the program can steal credentials, and how it covers its tracks so as to not be easily detected. Chapter 12 focuses on how Malware starts itself, and the variety of methods it uses, including process replacement and DLL manipulation so that it looks like any other innocuous running process. Chapter 13 goes into a variety of methods of encoding and decryption, while Chapter 14 covers ways in which the network is utilized to to take advantage of exploits.
Part 5 is Anti-Reverse-Engineering, and goes into the steps that malware developers go to to prevent their programs from being found or being cracked, Obscuring flow control, confusing dis-assemblers, prevent debugging, performing steps to use virtual machine detection and change the behavior or not run at all, and using methods to compress or encrypt files so that they are packed away until a given time or condition are all part of the toolkit of the malware developer to throw the malware examiner/tester off their tracks. we have to realize that while we are trying to outsmart the malware developers, they are trying to outsmart us as well.
Part 6 is a grab back of special topics including shell code analysis, special considerations for C++ code and how it differs from C, and some of the special challenges that the 64 bit processors and code base face and will face in the future.
The book rounds out with there appendices; describing important windows functions, a grab bag of tools that can be used fr malware analysis, and the biggest single section of the book, which is the answers and analysis to all of the labs provided.
This is a big book, and it covers a lot of ground. It is of course geared towards software developers, but there's a tremendous wealth of information for the software tester, too. While we may not spend as much time de-compiling or reverse engineering the code, having the ability to see the techniques that malware developers use, and the techniques that we can use to ferret out and stop them is enlightening. This book focuses on Windows and Windows exploits, and it's meant to be an introductory primer. It's not the be all and end off of malware analysis, but for many of us who are not specifically trying to be security experts, it's a heck of an introduction.
More than being a good introduction, this is a "get your hands dirty and do so without putting your self or your neighbors in peril" book. This is a dicey topic, and its one where, in the wrong hands, or handled carelessly, you can do a lot of damage to your self and to others, so take the time to set up, wall off, and sand box as much as you can. Proceed with caution, but if you do proceed, you couldn't ask for a better tour guide.