S-Video is audio only so you would need to connect the audio output of the camera to the converter audio input (if it has one).
S-Video carries the color information separately than the BW picture. Pins 2 and 4 carry that. You may have a bad cable, bad output of the camera, bad converter, or the pins aren't correct. 4 pin mini din cables are used for other applications and the one you got may not be compatible with S-Video although most are.