When comes to HiDPI screens and resolutions Firefox has always had some technical debts there. Wayland slightly improved it but we still miss clean user experience.
We tried hard to improve it and the last piece – hi-res widget rendering – landed in upcoming Firefox 68 (recently Beta). That means Firefox should be fully compatible with HiDPI screens and you shouldn’t see any glitches there.
Also Firefox 68 supports fractional scaling on Fedora 30 / Wayland which gives you more control over the screen resolution. To test it just grab Firefox Beta and try on your own, as:
MOZ_ENABLE_WAYLAND=1 ./firefox
Martin Stransky's Blog 