Upgrading ApiScout: Introducing ApiVectors

About a year ago, I published ApiScout, a library that allows the recovery of potentially used Windows API functions from memory dumps.

The approach can be outlined as checking all DOWRDs/QWORDs of a memory dump against a previously created collection of DLL information for a given Windows instance. It has been used in the recently published paper on Malpedia, where it was used to compare Windows API usage behavior of 382 malware families.

In this blog post, I want to explain the additions to ApiScout that I have done together with Steffen Enders, namely the introduction of ApiVectors. ApiVectors are a compact representation of “interesting” API functions extracted with ApiScout that can be used to get a first impression of a malware's potential capabilities but may also serve for matching against a reference database to aid in malware identification.

 

ApiVectors

The output of ApiScout is a list of tuples, consisting of the following elements:

• offsets:
• in the buffer where a potential API address has been found
• the dword identified as DLL/API address
• its membership in the PE header's Import Table (if applicable)
• an estimate how many times it was referenced in the code
• the found DLL / API name

Here is a shortened example result for a Citadel binary:

Example output of ApiScout for a dump of Citadel.

 

Naturally, we now wanted to find a decent way to store this information, preferably in a compact format that maintains as much relevant information as possible.

In our previous work on Malpedia we found that you encounter “only” around 4.000 unique APIs (out of 50k+ found in a Windows installation) across the 380 malware families, with very few being common (top150 APIs present in ~25% of the families) and many being found in just few families (90% of the APIs present in 10% or less of the families).

Of course, we could easily use the full set of APIs as base vector but that would mean super sparse vectors since we also found that on average, 120-150 APIs are found in a given malware family.

Because we wanted this vector to carry information useful to analysts, we had a closer look at the semantic context of the API functions. So we went ahead and labeled ~3.000 of the API functions into the following 12 groups (with counts per group):

• 584 GUI
• 392 Execution
• 353 String
• 312 System
• 278 Network
• 230 Filesystem
• 101 Device
• 088 Memory
• 073 Crypto
• 062 Registry
• 033 Other
• 022 Time

An immediate observation is that there are many API functions related to GUI and string handling that potentially are less interesting to an analyst, the same holds true even within more relevant groups. Instead of covering everything, our representation should definitely focus on really meaningful suspicious aspects like interacting with the system or network.

In our efforts for reduction, we first apply some simplifications to Windows API names:

1. We drop the string type, i.e. “A” or “W” if applicable
2. We ignore MSVCRT versions, i.e. msvcrt80.dll!time becoming msvcrt.dll!time

Some quick experiments showed that the information lost by this step is negligible.

We then went ahead and designed a custom vector of size 1024 that is roughly based 80% on the occurrence frequency as found in our Malpedia evaluations and 20% based on domain knowledge of interesting API functions that should definitely be included.

This leaves us with the following result:

Result of crafting a 1024 bit vector by semantic groups

 

The vector we have settled for can be found here. Feedback welcome! :)

 

Visualization: ApiQR

One cool thing that you can do with a vector of length 1024 is fitting it into a Hilbert curve to achieve a nice way to visualize the information. The Hilbert curve ensures that neighboured entries appear next to each other, while also filling the given space. We call these diagrams ApiQRs:

ApiQR representation: Hilbert curve for our 1024 bit ApiVector with the semantic categories

 

Here are some example visualizations for a few families. You can also click them to have their vectors viewed in Malpedia:

• RockLoader -
  A8gAgAFAIA3gA7IA4EAACA7CQA4QA8QABA3EA6FAEA5CA3IA69BAEAABAABA10
• TeslaCrypt -
  AAgAAgAAwAgAFQIAQMwysA5JA3gEQAgACA3BgCQLAAQACFCgIAIAELJBIA5QBL}AEAiA3GAEQA58EgIASwA6BABAABAgA6QAKIA
• Citadel -
  ErkAAQIABBpe,QIgUIwytA5JzhgAOYAm.aBhMwBq]Ot+twChA,]gIABLBAYCiVMswMAZACAA?BjBEIQAAgAIwj-aBCQNA5QAIAD,-eQoiQkQmBAH]DA3ngACAAQAEEAALI-kAE^acSzABCAABhSBEAREgBhgEQCXoYI+
• DarkComet -
  AARogzQx/,wIiQcwCuABj.RJDlujHYQgv*AC,_fe@LO-}TIcIA3IasLJBgCAntMwNLFAEAAYhlADAQABgEAxAQCAAQAAIA3wAMAz_kYitAQhAFA10BD_^/fDMBn.-_mADwgASgDTkAAEgPFNDHAgbBhIAGToaKQ

RockLoader

TeslaCrypt

Citadel

DarkComet

 

Compact Representation of ApiVectors

 

You may be interested how the above base64-like strings (e.g. for RockLoader: A8gAgAFAIA3gA7IA4EAACA7CQA4QA8QABA3EA6FAEA5CA3IA69BAEAABAABA10) are constructed.

We actually use an alphabet of 74 printable (and hopefully not too tool-conflicting) characters in a way that is actually very similar to base64.

Our custom base64 alphabet has the characters:  

"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz@}]^+-*/?,._"

and we add the 10 numbers to later allow us to use runlength-encoding.

Similar to base64, each character of the alphabet encodes 6 bits of the vector and the mapping is the following:

'000000' -> 'A', '000001' -> 'B', '000010' -> 'C', '000011' -> 'D',
'000100' -> 'E', '000101' -> 'F', '000110' -> 'G', '000111' -> 'H',
'001000' -> 'I', '001001' -> 'J', '001010' -> 'K', '001011' -> 'L',
'001100' -> 'M', '001101' -> 'N', '001110' -> 'O', '001111' -> 'P',
'010000' -> 'Q', '010001' -> 'R', '010010' -> 'S', '010011' -> 'T',
'010100' -> 'U', '010101' -> 'V', '010110' -> 'W', '010111' -> 'X',
'011000' -> 'Y', '011001' -> 'Z', '011010' -> 'a', '011011' -> 'b',
'011100' -> 'c', '011101' -> 'd', '011110' -> 'e', '011111' -> 'f',
'100000' -> 'g', '100001' -> 'h', '100010' -> 'i', '100011' -> 'j',
'100100' -> 'k', '100101' -> 'l', '100110' -> 'm', '100111' -> 'n',
'101000' -> 'o', '101001' -> 'p', '101010' -> 'q', '101011' -> 'r',
'101100' -> 's', '101101' -> 't', '101110' -> 'u', '101111' -> 'v',
'110000' -> 'w', '110001' -> 'x', '110010' -> 'y', '110011' -> 'z',
'110100' -> '@', '110101' -> '}', '110110' -> ']', '110111' -> '^',
'111000' -> '+', '111001' -> '-', '111010' -> '*', '111011' -> '/',
'111100' -> '?', '111101' -> ',', '111110' -> '.', '111111' -> '_'

 

Now, take the following raw, uncompressed ApiVector of RockLoader shown above and the corresponding encoding per 6 bit below each row:

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A

100000000000100000000000000101000000001000000000

g     A     g     A     F     A     I     A

000000000000100000000000000000000000000000000000

A     A     g     A     A     A     A     A     

000000000000001000000000000000000000000000000100

A     A     I     A     A     A     A     E

000000000000000010000000000000000000000000000000

A     A     C     A     A     A     A     A

000000000000000010010000000000000000000000000000

A     A     C     Q     A     A     A     A

010000000000000000000000000000000000000000000000

Q     A     A     A     A     A     A     A

000000010000000000000001000000000000000000000100

A     Q     A     B     A     A     A     E

000000000000000000000000000000000000000101000000

A     A     A     A     A     A     F     A

000100000000000000000000000000000000000010000000

E     A     A     A     A     A     C     A

000000000000001000000000000000000000000000000000

A     A     I     A     A     A     A     A

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000000000000000000000000000000000000000000000000

A     A     A     A     A     A     A     A     

000001000000000100000000000000000001000000000000

B     A     E     A     A     B     A     A

000001000000000000000000000000000000000000000000

B     A     A     A     A     A     A     A

0000000000000000(00)

A     A     A

(Small remark: Obviously we have to pad with 2 bit because 1024 % 6 == 2.)

Now, one thing that is obvious is that vectors can be very sparse and we can probably condense the representation further.

For this we use runlength-encoding, with which we can remove the repetitive consecutive symbols, for which we freed up the 10 numbers from the original base64 alphabet before.

With that, we can now “compress” the vector as follows.

Uncompressed: AAAAAAAAgAgAFAIAAAgAAAAAAAIAAAAEAACAAAAAAACQAAAAQAAAAAAAAQABAAAEAAAAAAFAEAAAAACAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAEAABAABAAAAAAAAAA

Compressed: A8gAgAFAIA3gA7IA4EAACA7CQA4QA8QABA3EA6FAEA5CA3IA69BAEAABAABA10

This is also the final representation.

Here are some statistics, taking malpedia as basis (1957 dumps for 637 distinct families):

Avg number of unique APIs per dump: 121.633
Avg number of APIs represented in ApiVector: 100.053 (82.258% coverage)
Avg ApiVector length (compressed): 92.438 bytes

Compression Rate (vs. 172 bytes uncompressed): 1.86x

Okay, but how can I use this?

You can easily incorporate ApiVectors into your own analysis environment.

For starters, the previous blog post on ApiScout explains how to build a DB custom to your Windows system. After this, you can simply crawl arbitrary buffers (e.g. memory dumps of selected suspicious segments from processes) for their API information and have this available in your other analysis such as IDA Pro.

If you do not want to use ApiScout to crawl memory dumps, you can also create ApiVectors directly from a given list of Windows API functions (e.g. Import Tables) using getApiVectorFromApiList() and getApiVectorFromApiDict() from the ApiVector class respectively.

A concrete use case for ApiVectors is matching them against each other. 

In that context, projects like ImpHash and ImpFuzzy may come to mind.

The advantage of ApiVectors is that they actually carry the identity of API functions used without abstracting them with only little higher cost in terms of required storage. We are currently looking to hook our approach up with sandboxing, e.g. Cuckoo.

Our current experiments indicate that using similarity of ApiVectors may in fact serve as a decent way to perform malware family identification.

As announced, we will cover this in a future blog post and full paper summarizing all of our findings around ApiScout.

ApiScout: Painless Windows API information recovery

After hacking away for some days in the code chamber, I'm finally satisfied with the outcome and happy to announce the release of my new library: "ApiScout".
The main goal of ApiScout is to allow a faster migration from memory dumps to effective static analysis.

While reverse engineering "things" (especially malware), analysts often find themselves in a position where no API information is immediately available for use in IDA or other disassemblers.
This is pretty unfortunate, since API information is probably the single most useful feature for orientation in unknown binary code and a prime resource for recovery of meaning.
Usually, this information has to be recovered first: for example by rebuilding the PE ("clean unpacking", using ImpRec, Scylla, or similar) or by recording information about DLLs/APIs from the live process to be able to apply it later on (see Alex Hanel's blog post).

Both methods are potentially time-consuming and require manual effort to achieve their results. From my experience, clean unpacked files are often not even needed to conduct an efficient analysis of a target.
As I did a lot of dumping when reversing malware over the last years (and especially for malpedia - project outlook slides here), I craved for a more efficient solution.
Initially, I used a very hacky idapython script to "guess" imports in a given dump versus an offline DB - the limitations: 32bit and a single reference OS only.

After talking to some folks who liked the approach, I decided to refactor it properly and also integrate support for 64bit including ASLR.

TL;DR (Repository): ApiScout

To show the usefulness of this library, I have written both a command line tool and IDA plugin, which are explained in the remainder of this blog post.

First, let's have a look at a more or less common situation.

A Wild Dump Appears

For the purpose of illustration we use 1e647bca836cccad3c3880da926e49e4eefe5c6b8e3effcb141ac9eccdc17b80, a pretty random Asprox sample.

Executing it yields a very suspicious new svchost.exe process.

Running the Asprox sample results in a new suspicious scvhost.exe process.

Inspecting the memory of this new process reveals a not less suspicious memory section with RWX access rights and a decent size of 0x80000 bytes.
However, apparently the PE header got lost as can be seen on the left:

Looking closer at the process memory, we find a RWX segment @0x008D0000.

Luckily the import information is readily available:

Left (Hex view) /Right (Address view): Import Address Table (IAT) as found inside of the RWX segment.

With ImpRec or Scylla, we would now have to point to the correct IAT instead of using the handy IAT autosearch, because autosearch would identify the IAT of svchost.exe instead of Asprox' (see comparison left vs. right).

Left: Scylla IAT Autosearch gives IAT of svchost.exe, but we want ...
Right: IAT of Asprox - which we can't dump since PE header is missing.

But we now encounter another issue: Because there is no PE header available, Scylla fails to rebuild the binary and with that, the imports.
Granted, many injected memory sections will have more or less correct PE headers or we could write one from scratch...
But remember, I promised "painless" recovery in this blog post's title.

ApiScout: command-line mode

As I explained before, if we have all relevant API information available, we can directly locate IATs like the one of the above example.
So let's first build an API DB:

Running DatabaseBuilder.py to collect Windows API information from a running system.

While DatabaseBuilder.py is fully configurable, using Auto-Mode should yield good results already.

Next we can use the database to directly extract API information from our dump of memory section 0x008D0000:

Resultof running scout.py with the freshly build API DB against a memory dump of our injected Asprox.

Since this cmdline tool is just a demo for using the library, this should give you an idea of what can be achieved here.
For our example memory dump (76kb), I timed the full recovery (loading API DB, searching, shell output) on my system at about 0.3 seconds, so it's actually quite fast.

I am aware that this may occasionally lead to False Positives but there is also a filter option as a simple but effective measure: It requires that there is at least another identified API address within n bytes of neighbourhood - from my experience this is already enough to reduce the already very few FPs to an absolute minimum.

IDA ApiScout: fast-tracking import recovery

In this section, I want to showcase the beautified version of my old hacky script.
I assume it can be similarly adapted for others disassemblers like radare2, Hopper, or BinaryNinja.


Loading ida_scout.py as a script in IDA shows the following dialog in which an appropriate API DB can be selected.
Note that imports are not resolved as we loaded the memory as a binary (not PE) at fixed offset 0x008D0000:

ida_scout.py shows the available API DBs or can be used to load a DB from another place.

Executing the search with the WinXP profile from which Asprox was dumped, we now get a preview of the APIs that can be annotated:

Selection/Filter step of identified API candidates.

Aaaaand here we go, annotated API information:

Yay, annotated offsets in IDA as if we had a proper import table!

And yes, it's just as fast as it seems, clicking through both windows and having API information ready to go took less than 10 seconds.

That's what I call painless. :)

Dealing with ASLR

For simplicity's sake the above example was executed on WinXP 32bit, with no ASLR available.
However, it works just as fine for more recent versions (I use Windows 7 64bit), both for 64bit dumps or 32bit compatibility mode dumps.
In case you haven't disabled ASLR on your reference system, this section explains how ASLR offsets are obtained for all DLLs that are later stored in the DB.

I will skip explaining ASLR in detail, but feel free to read up on it, e.g. this report by Symantec.

The first step of DLL discovery is identical to non-ASLR systems and performed by DatabaseBuilder.py.
At the end of the crawling process (which involves collecting the ImageBase addresses as stated in the PE headers of all DLLs), we perform a heuristic check if ASLR is activated: We obtain a handle (which equals the in-memory BaseAddress) to three DLLs (user32.dll, kernel32.dll, and ntdll.dll) via GetModuleHandle() and check if the respective corresponding file as identified with GetModuleFileName() shows an identical ImageBase. If at least one DLL differs, we assume ASLR is active.

Since every DLL receives a individual ASLR offset, we will have to make sure that every DLL of interest has been loaded at least once.
For this purpose, I wrote a little helper binary "DllBaseChecker[32|64].exe" which simply performs a LoadLibrary() on a given DLL path and returns the load address.
Iterating through all DLLs identified in the discovery step, we are now able to determine each individual ASLR offset by subtracting file ImageBase and load address.

Closing Note

While this approach probably is certainly no magic or rocket science, I haven't seen it published in this form elsewhere yet. At least to me, it provides great convenience in several ways and I hope that one or the other can benefit from it as well.

For future use, I imagine it being used manually as shown in the post or potentially in automated analysis post-processing chains, where this functionality may come in handy.

I have to admit that I misjudged the effort to do code this in a nice way (by about a week of release-time) but I want to thank @herrcore for motivating me to rewrite and release it and @_jsoo_ for pushing me to address ASLR properly with the initial release version.

Code is here: ApiScout


As I want this to become a tradition: this blog post was written while listening to deadmau5's new album "stuff I used to do". :)