I use the Ortleib quick rack since I use my bike for regular gravel riding too. Takes roughly 30 seconds on and off. Only leaves 2 small mounting points on the chainstay which is awesome.
No issues in the year I used it for commuting with two 10 lb bags on each side. If you have big feet you may hit your bags in certain configurations though. Have to be mindful of how you mount the panniers.