Also, historically KDE4 had a lot of memory leaks. I haven't used Plasma 5 enough to comment, but it isn't exactly lightweight.

Actually, Plasma now consumes as much ram as XFCE. It's pretty lightweight tbh. It consumes around 400MB on boot while GNOME consumes 600MB.

But the thing is that memory consumption is not a good way of measurement software quality.
Not used RAM is wasted RAM. I prefer a software that consumes more ram but its more responsive and stable than the other way around.
Also, Linux is pretty good at managing it and keeping up the balance.

I was going to suggest running latencytop, but it has been quite a while since I have used it, and I just discovered that Ubuntu 20.04 does not have CONFIG_LATENCYTOP set in its stock kernel.

You could try to look at all the different metrics under heavy load during gaming. Just use MangoHUD, and you'll see real time info.