Trying to be helpful even if i may not know much about encoding or codecs, but i know that "MPEG4/H.264" also known as "AVC" compression standard is what is used by all the highest quality video files i have seen (except of course for completely uncompressed). The files also tend to use the Matroska (mkv) format, although i'm not sure what the benefit of that is, but when i knew less about the subject when i was looking for quality video files i sniffed out the ones in the mkv format for the best results a few years ago.
The profile the very best quality files tend to use is (quoting wikipedia)
High 10 Profile (Hi10P)
Going beyond typical mainstream consumer product capabilities, this profile builds on top of the High Profile, adding support for up to 10 bits per sample of decoded picture precision.
Also known as 10-bit Encoding. However a capable media player is required to play these files and from what i've seen only the most up-to-date media players are compatible with the profile (i haven't seen a video editor that can handle it yet, however i haven't tried many) players that support it include Media Player Classic (with the right codec pack, i use K-Lite), Mplayer2 and VLC.
Use 10-bit encoded H264 files for maximum quality vs compression rate. Also to be noted is that 10-bit encoded files tend to be smaller than the traditional 8-bit ones. I can also guess that changing the resolution of the video may cause this effect.
To be noted is the Hi444P profile which supports 14-bit encoding, however i have not seen it in use and because of that i do assume that the support for that is extremely limited.
That's all the advice i can give on the subject of video quality as a quality freak (as a "quality freak", the media player/renderer i prefer is Media Player Classic with MadVR renderer and LAV filters, my alternative on linux is Mplayer2 with OpenGL renderer, i refuse to touch VLC and consider it little more than an ugly relic with decent amount of codec support).
Update:
If this is correct the 10-bit encoding i mentioned is what is needed to get rid of the artifact. I wonder why the effect is in the wallpaper though.. did you take the screenshot from a video rather than actually in-game?
This is where the cat is from and yes you should definitely watch that.