Analyzing the SecuROM 8.10.X VM.

I want to thank ARTeam for providing the docs on SecuROM 7.30 VM they really helped and are mostly still relevant today.

That said, I have not worked on SecuROM 7.30 ever, but I believe the VM has changed since then.

Here is an overview of the VM initialization.

When entering the VM, an argument is pushed to the stack. It's a pointer to a pointer to the VM opcodes. I call this argument a "program".
The dummy call after pushfd is used to get the address of the VM.

In the picture above several things happen. A spinlock is created by the thread which enters the VM and will initialize the context, all other threads, if any, will wait till the first thread has finished the initialization.

Then SecuROM uses the loop x86 construct to loop over all 100 possible VM thread contexts, and finds the first free one. The busy flag 0x66666666 indicates if a thread is busy or not. I should note that SecuROM 7.30 only supported up to 10 threads, SC 8.10 supports 100.

After the first free context is found, SecuROM jumps to the following code which sets the busy flag.

lea edx,[ebx+24]
mov dword ptr ds:[edx],66666666

then the VM context is zeroed out, but care is taken not to zero out the busy flag. Afterwards the lock is removed and the jump "je short 38D702FE" takes us to the last step of the initialization.

pop ebx loads the VM context for this thread in ebx.

sub dword ptr ss:[esp],7 subtracts 7 bytes from the VM function address which I mentioned above that it is pushed to the stack with a dummy call so it ends up as 38D70280 in this exe.

This part fills the VM context struct. I've taken the liberty of adding captions next to the instructions which are self-explanatory.

Next is this obfuscated code.

In a nutshell, it fetches the delta to the pointer to the opcodes, adds the VM entry point, and fetches 2 DWORDs(aka 8 bytes).

CPU Disasm
Address   Hex dump                 Command                                                      Comments
38D7035A    8B70 04              mov esi,dword ptr ds:[eax+4]
38D7035D    8B00                   mov eax,dword ptr ds:[eax]

The first 4 bytes of the opcode is the modifier, the next 4 bytes is the obfuscated address of the handler.
The modifier is used to calculate the next handler address. The one in the VM context is updated with this new one, after some xor and shifts are performed.

The "encrypted" address of the first handler is decrypted with a XOR.

xor esi,48371826

 It's then copied to eax.

Finally

CPU Disasm
Address   Hex dump                 Command                                                      Comments
38D70396    B9 19000000       mov ecx,19
38D7039B    83F1 1D              xor ecx,0000001D
38D7039E    01D9                   add ecx,ebx
38D703A0    8301 08              add dword ptr ds:[ecx],8 <-- Add 8 bytes to VM EIP
38D703A3    FFE0                  jmp eax

The VM EIP(program counter) is incremented by 8 bytes, and we jump to the address of the first handler.

Here comes the juicy part. The jump to first handler goes to this code

Step 1.

First, ebx+20 is updated with the address of the next pseudo handler, for a lack of a better word. And if we follow the jump we end up where the actual first instruction is executed in this particular handler.

Step 2.

and if we follow the jump we end up at what I call the "dispatcher". 

Step 3.

The dispatcher adds the VM entry point to the value added in ebx+20 to form the address of the next pseudo handler.

Basically, from what I understood, a single handler which is usually a sequence of instructions has been split into several small pseudo handlers each reached in three steps. In my opinion this is just obfuscation to slow down reverse engineering.

Sometimes in Step 1 there is an additional instruction that moves a value in ebx+400, which is usually used to substract 4 bytes from the stack pointer.

Now, if we follow the each jump and where it leads to, remove all jumps and dispatcher code, the first handler's code is basically this.

mov esi,dword ptr ds:[ebx+4]
add esi,dword ptr ds:[ebx+0C]
add esi,4
push dword ptr ds:[esi]
pop edi
mov dword ptr ds:[ebx+400],4
sub esi,dword ptr ds:[ebx+400]
push dword ptr ds:[esi]
pop esi
mov cl,byte ptr ds:[ebx+10]
push eax
xor eax,esi
xor eax,dword ptr ss:[esp]
add esp,4
shl eax,10
shr eax,18
xor al,2A
add byte ptr ds:[ebx+10],al
mov eax,3BF7C894
push edi
push eax
mov eax,esi
shl eax,18
shr eax,18
ror al,cl
xor al,78
shl eax,2
add eax,ebx
mov edi,eax
pop eax
push eax
pop dword ptr ds:[edi]
pop edi
push edi
pop eax
sub al,cl
xor eax,00B42D00
add dword ptr ds:[ebx+4],8 <-- Update VM EIP with 8 bytes.

In my case, this handler basically calculated the address of another handler.

This isn't an exhaustive analysis of the Securom 8 VM, it's but a scratch of the surface. Furthermore, I have not identified the VM exit procedure.