It started out occasionaly, but now we never roll it down as it can take up to 10 minutes of fiddling with the switch to get it to go back up.

Things that come to mind first are the switch, a loose wire, or a dead spot in the motor that is in the right spot when it stops in the down position.

Another forum mentioned something about there being a relay in the door causing the window to go down on it's own after turning the engine off, then starting it again (not my issue but possibly the same part misbehaving).

I thought I'd ask before tearing things apart just in case someone had the issue as well.
I'm also waiting for warmer weather.

 

When I removed the switches and swapped them around, the problem went with the switch, so that is the problem.