TimRiker,
Lately I've been dealing with this same issue on both of our factory rearview mirrors on the Silver Kitten (i.e. our 2002 E-350 SM

. I have developed a what seems like a reasonable fix for mirror wobble that I've been testing for several months now. I'll try to describe it briefly here, along with my process for removing the mirror element, in the hope that this may help. Eventually, I will finish my full write-up on the process with pictures and post it here on the SMB forum, but my family and work obligations must come first, for now.
Mirror Element Removal/Replacement:
1) Looking at your second picture, which looks into your driver's side mirror from the outside edge, look just to the right of the black rubber boot which surrounds and protects the movement mechanism for the mirror. If you look carefully, you will notice that the plastic mount on the back of the the mirror element consists of a segmented ring of "fingers" that latch onto a circular mount on the moveable part of the mirror-positioning mechanism.
2) I moved my mirror all the way in one direction, just as you have done in this second picture, and then reached inside of the mirror housing with a tiny auto-body-trim right-angle pry bar (5-inch??? i.e. almost small enough to fit in the palm of my left hand), and sequentially lifted up each of the plastic fingers on the back of the mirror enough for it to release - all while prying outward on the glass (i.e. towards the rear of the vehicle) by pulling GENTLY on the edge of the mirror with the thumb of my right hand.
Note: Keep a hold of the mirror element with the thumb and fingers of your right hand (don't worry, you can clean it later) so that the mirror element won't fall and break when it suddenly pops free from its mounting. Having a sleeping bag or under-inflated air mattress spread out on the ground underneath the mirror is also a reasonable precaution in case you drop the mirror element when it comes free from the mechanism.
3) Working my way around the mirror element, I kept a constant and GENTLE pull on it with my right thumb while sliding my thumb around the rim of the mirror element, gradually tilting the mirror element to its other limits of movement (i.e. Down, Left, Up). Don't worry, you won't hurt the gear-motor mechanism that moves the mirror by moving the mirror manually.
4) When you get a little over half-way around the mirror, prying the fingers up one- or two-at-a-time as you go (yeah, you'll probably have to back up and re-do a few of them too), the mirror element will eventually just pop off of the mechanism once you've lifted enough of the fingers in a row, and kept them from snapping back into place.
5) To reinstall the mirror element, you can just square up the mechanism with the housing, line up the mirror element with the housing, and gently press evenly on the center portion of the glass, and it will snap back into place.
Note: Removing the mirror element for the second time is much easier than the first!
"The Mirror Wiggle Fix"
As for the wiggle, the pivot mechanism in these mirrors relies on precision-molded plastic parts that inevitably wear with time, and I have found little evidence of an effective lubricant being used. (Maybe it all evaporated during the original owner's time with our SMB?) And once the plastic bearing surfaces wear, the slop and shake are almost unavoidable.
However, I have taken the slop and wiggle out of our passenger-side mirror by using one or two modified throttle return springs (Dorman Part #29005) from the "Help!" section of my local auto parts store. The spring(s) can hold the moveable part of the mirror mount snug against the rest of the mechanism, without being too strong for the motors to move the mirror, and without causing the mirror to drift out of position over time under off-road vibration.
It really helps!
You have to cut and modify the spring(s) a little, and then drill a few tiny mounting holes in the plastic movement mechanism for the ends of the spring(s) to hook into, but it seems worth the trouble to achieve a $5 fix on a power mirror!
I'll try to write this process up with pictures as soon as I can . . . but for now, I've gotta run!
Timerider
P.S. If I find a better tool for doing the mirror element removal that can be cheaply bought or easily made, I'll include that in my write-up as well. Maybe other more-experienced forum members have already developed better tools and processes for doing this . . . I dunno. If so, hopefully they will post or link to them here.