I dug a little deeper into the winch problem today. First I disconnected all the wires from the solenoid except the direct from the battery lead and tested the solenoid. The solenoid has 4 posts on it. As far as I can tell the solenoid is acting normal with no wires hooked to the motor. When line in is selected post B becomes hot. When line out is selected post C becomes hot.
Once I connect everything back up it only runs in line out mode. In line out mode post C is hot and post B stays neutral. When I select line in mode all of the terminals become hot and the motor does not spin. Only one of the two directional leads should be hot at a time. In line in mode both of the directional leads are hot. To me this is an indication something is shorting out inside the motor allowing current to pass back through both the line in and line out leads.
I pulled the winch and took the end cap and housing off. There was nothing in there screaming foul to me so I put it all back together being careful to reset the brushes in the proper position.
I'm not going to reinstall the winch till I talk to superwinch on Monday and see if they have any words of wisdom for me.
I couldn't swap the two directional leads due to the length of each lead. There wasn't enough freeplay in the cables to cross them over.
This is a shot of the solenoid and winch motor wiring. The lead from the solenoid to the winch motor on picture right is from post C or the line out side. The longer lead that loops out to picture left is from post B or the line in lead. Post A on the solenoid is the direct to battery lead and post D is the short cable whose function is a mystery to me. The ground cable is on the bottom of the motor housing and you can't see it in the picture.