For some reason, I'm unable to make it work with anything other than window capture mode and my game has to be in windowed mode. Unfortunately my games becomes very laggy after trying to stream and remains laggy even when not streaming anymore so I'm kind of in them midst of trying to see what solution I can come up with. Running a 1080 and never experienced this issue until I tried streaming :(.
Also, make sure the source for the game capture is at the bottom and try resetting under transformation. I think that helps me when I was having something simliar and ensure your resolution is properly configured.
English