[THOUGHTS/EXPERIMENTAL] Virtual Function Resolver Considerations #18
Description
We are in the rather special position that SHC does not seem to use any inheritance for its custom logic.
Regardless, I originally started to brainstorm some ideas that may allow applying a system similar to the current FunctionResolver to virtual functions.
However, further research revealed that I was rather mistaken, and that there is barely any way around the virtual function tables once it comes to virtual functions.
Since we do not need them, this text is only documentation, and might later move to the website should someone ever wonder if our approach could be used for games using VTables.
Naturally, this means nothing was tested, and this ideas and research might turn out to be wrong.
The nature of a member function pointer depends heavily on the compiler. MSVC luckily only uses simply addresses for non-virtual member functions.
However, it switches to a combination of offsets and addresses once if becomes virtual and inheritance is involved.
The current function resolver can therefore only handle non-virtual member functions of MSVC, since is needs to be possible to handle and cast the external function address from an integer, while also getting the pointer to the own function to create the proper detours.
All of this falls flat with virtual functions, since they take the actual address (most of the time) from their vTable, properly linked via a pointer inside the object (most of the time, since the standard does not know them).
This moves the core of the issue to the vTables. If we reimplement a class hierarchy, we automatically create virtual tables the moment we add virtual functions. This means we would have two virtual tables of distinct structures meant to partially replace each other.
This means the following things:
- It becomes a V-Table issue. Virtual functions do not need to resolve to different function pointers on usage, since objects search for the pointer linked inside them. The member pointer likely only tells them how to resolve this pointer (or not at all, in case of MSVC non-virtual).
- The Vtables need to be synced. Basically, no matter who controls the objects, game or reimplementation, both objects would need to end up at the same function.
- To achieve the latter, the compiler behavior would need to be researched. If it just encodes the order in source, it becomes a lot easier. If not, a lot harder.
- It might also be possible to replace the pointer to the vTable inside the hooked games constructor with the one to the own table, but then it needs to happen before any object was created by it.
- The resolve for them might target classes, syncing and overriding their tables, not their virtual functions directly. However, if the structure is predictable, it may also happen partially.
- Pointers to the tables would need to be dynamically resolved, which usually happens by creating an object and getting the table from it. This might be problematic in case of big singletons. But at least it would happen on initialize only.
- Multi-inheritance would be less "easy". One can only hope it is not used. As a pattern it is discouraged anyway.
- It may be possible to extract information from the actual member function pointer, but this will require research on its structure.
- Wrapping might involve placing a function with a fitting call pattern inside the table instead of the function to use.
- To properly create the tables and allow the detouring, it might be required to always implement the functions in some way, so leaving the undefined might not be possible anymore.
- At least windows uses RTTI (Runtime Type Information) for dynamic_casts. If they are used, they might also need to be synced. Usually the are an address above the vTable. Since they often resolve to a call if virtual functions are used, it might be needed to hook this helper function in general.
As a conclusion, it is certainly still possible to achieve a resolver like behavior, but the approach would differ a lot from the other resolvers.