Need some advice about creating game overlay in Linux
Need some advice about creating game overlay in Linux
Hi. I've been learning Rust for a while, and I want to take on an actual project now to learn even more. I need to be pointed in the right direction for one aspect of the affair, and I hope someone here can help me.
I want to create a deck tracker for Hearthstone that runs natively in Linux. This is, on the back end, a fairly simple matter of parsing a constantly updated file that tracks everything that happens in the game. On the front end, however, I want to create a window that sits on top of the fullscreen Hearthstone window and shows me stuff. The "stuff" doesn't have to be images or anything fancy, I'll take whatever I can get, but I don't know how to get started on this part.
So the task is as follows: Create an overlay on top of the fullscreen Hearthstone client, preferably under Wayland, and update it constantly with new information about cards drawn, cards left in deck, that sort of thing.
How do I tackle this problem? Are there any crates that'll let me create such a window and render stuff to it? How would you approach the problem?
Thanks in advance.
There's two main options to approach this: either you inject into the game like how MangoHUD does it, or you make an overlay window that your window manager/compositor helpfully places on top for you.
The second option is pretty simple as any GUI library will give you the window, you just need some help from the compositor side to do the rest. That part can also be pretty easy thanks to Wayland nesting: you can use whatever compositor you want for this, not just the one from your DE. Gamescope for example might work, if not wlroots+layer-shell.
You can also be your own nested compositor by using Smithay, which is in Rust and was made by System76 to be the library behind COSMIC's compositor. Looking at the examples, it looks not too crazy to use. Then you just run the game under yourself and you can do whatever you want.
Injecting into the game isn't too crazy either, you compile to a library and force load it with LD_PRELOAD and override some OpenGL/Vulkan functions that lets you add your own draw commands on top before the frame is finished.
I think avoiding injecting into the game is cleaner and easier as you're completely independent from the game client, so you won't crash the game and you can also just test out your code as a regular window, and restart it mid game. Injecting would also trigger anticheat if present. Which, what you're doing kind of is, even though anyone could also just pen and paper it.
Thanks so much for taking the time to write that long post. I have lots of things to dig into now. I think I'd prefer not injecting anything into the game for the reasons you mentioned, the most important consideration being the anticheat risk. I don't know what the company behind the game have implemented of that nature.
I'll preface this by saying that I'm not familiar with Rust nor Hearthstone at all, but I do deal with D3D9 and D3D11 on Windows to do similar things. Hopefully this will give you insights how you could approach this. (Closest I've done was code injection on Android)
The most common and robust approach to this is to hook/detour the API functions that the game imports from the renderer backend.
One way you usually do this is by creating a dummy library which overrides/intercepts the system library and passes through every function call to the API, except for the ones you need, you'd put your code before/after the passthrough. This usually requires you to gather all exported symbols and re-create them, which is a very tedious but rewarding task, as it usually is very stable and can work around things such as DRM.
Usually, since that sits quite low on the application's code stack, it is most efficient for it to be a more general-purpose hook which can load other libraries. Examples would be things like the ASI loader or Reshade on Windows.
Another way would be to do code injection via library side-loading. Essentially, you can simply load a library that performs the code hooks and does necessary renderer API hooking. This is usually done in combination with the previous method (it being a "plugin" loader), however, it is also possible to modify game binaries to call
dlopento load your library and its exported function as an entrypoint (in which case you need to do platform's CPU assembly code for a bit).Those are the entrypoints for your code. After that, it is all about necessary render backend code that you need to do in order to draw graphics/text/etc.
In C/C++ land I'd just tell you to use Dear ImGui, but seeing as that doesn't exist for Rust, you're kinda on your own.
Same with the API detouring. Ideally, you'd make a plugin loader that does the job for you. Not sure if that exists in Rust yet.
For references, Vulkan overlays such as MangoHUD or ReShade could be useful to help you figure out how to draw stuff on screen.
As for the rest of your code - it can run in a separate thread that does the job for you as the game runs. Or, make a client-server relationship and make the game hook be the server for your info that you need.
Dear imgui absolutely exists on rust. Which is the same dear imgui as C/C++. If you want a rust option though, there's egui, which is basically Rusty's version of dear imgui.
Oh cool, I didn't know!
I'll go check it out, thanks!
I want to try to use Rust myself as well to build a library and I wonder how it'll turn out (especially since I do Win32 hacks mostly lol).
Thanks for your detailed reply. I can feel I'm out of my depth in many ways, but between your reply and the others I've gotten, I have a lot of entryways into the problem, and I'm looking forward to figuring out how to make it work. I've done a bit of coding in C++ in the past as well; maybe that would also be an option. But since the purpose of the exercise is primarily to get more familiar with Rust, I think I'll exhaust whatever options I have down that path first. Thanks again :)
No problem. Please do report back!
We really do not have many (if any) Rust alternatives for code hooking and injection and doing stuff with rendering like we do with C++.
Maybe you could finally figure something out we can use for other games as well!
Maybe someone can correct me if I'm wrong, but doesn't Wayland's security model intentionally prevent this type of "tampering" with another application's display?
It doesn't by default and design, but that doesn't mean it can't be implemented through a special protocol or compositor plugin.
This also protects against windows, not processes of the same user. You can bypass the problem by simply wrapping the game and your overlay in a nested compositor like gamescope. From there you control the compositor, so you can do whatever you want.
And it's still secure because it only lets you overlay over stuff from your own Wayland clients spawned by your overlay wrapper, none of the user's other windows, so you can still trust your password manager and such.
OP here. Thanks for your reply. If what you're asking is the case, I'd be happy to find a solution that runs under X instead. This tracker would in any case mostly be for my own use. I was just excited to finally get Wayland working with my NVIDIA card with the explicit sync stuff.