Are you sure it's not a bad connection in the speaker itself - that as you move the connector it makes or breaks?
I think I'd try swapping speakers left/right to make sure the problem stayed with the wiring harness and not the speaker.
I think I'd try swapping speakers left/right to make sure the problem stayed with the wiring harness and not the speaker.