Well, the boss can get a bit nasty if you're not doing that baiting thing with him. But the main danger from him comes only into play when you're actually focusing on other enemies.
The boss isn’t the annoying part for me.
Once you get the spawn timing and patterns down, you can control the flow of the fight.
It’s just the shield knights that are overkill