c - "Trapping" a processes' own sysenter calls in userspace on Windows -

i'm working on runtime non-native binary translator in windows, , far i've been able "trap" interrupts (i.e. int 0x99) os binaries i'm trying emulate using ugly hack uses windows seh handle invalid interrupts; because system call vector different 1 in windows, allowing me catch these "soft" exceptions doing this:

static int __stdcall handler_cb(exception_pointers* pes, ...) {      if (pes->exceptionrecord->exceptioncode != exception_access_violation)         return exception_continue_search;      char* instruct = (char*) pes->contextrecord->eip;      if (!instruct)         handle_invalid_instruction(instruct);         switch (instruct[0])     {         case 0xcd: // int         {             if (instruct[1] != 0x99) // int 0x99                 handle_invalid_instruction(instruct);             handle_syscall_translation();             ...         }         ...         default:             halt_and_catch_fire();     }     return exception_success; } 

which works (but slowly), problem windows first attempts handle instruction/interrupt, , non-native binaries use sysenter/sysexit instead of int 0x99, systenter instructions in non-native binary valid nt kernel calls when executed, meaning handler never called, , worse; state of "host" os compromised. there way "trap" sysenter instructions in windows? how go doing this?

as far know, there no way (from user-mode process) "disable" sysenter, executing generate exception. (i'm assuming programs don't try sysexit, because ring 0 can that).

the option think have virtualbox does, , scan invalid instructions, replacing them illegal opcodes or similar, can trap on, , emulate. see 10.4. details software virtualization.

to fix these performance , security issues, virtualbox contains code scanning , analysis manager (csam), disassembles guest code, , patch manager (patm), can replace @ runtime.

before executing ring 0 code, csam scans recursively discover problematic instructions. patm performs in-situ patching, i.e. replaces instruction jump hypervisor memory integrated code generator has placed more suitable implementation. in reality, complex task there lots of odd situations discovered , handled correctly. so, current complexity, 1 argue patm advanced in-situ recompiler.