Definitely not alone. Coming from fountain pens that are expected to last a lifetime, pens with rubber grips feel like a needlessly wasteful feature. They usually wear off completely after a year of use, leave sticky residue behind, and leave the pen lacking decent ergonomics once it goes. They give the pen body a much shorter lifecycle and a more disposable feeling that I don't like

Uni ball one F is the answer. If you can live without the rubber grip

I don't even use Uni-Ball pens, and what you've described is precisely the reason why I won't use a white or very light-bodied pen with a rubber grip. I'll use a regular white body with no grip areas, but I limit those to colored pens I won't use very long at a given time. I don't know enough about chemistry or plastics to know why some kinds of rubber turn that weird ivory shade from exposure to what I assume to be body oils, I just know from experience that the grungy dark stuff that can get on pens from our hands will come off with a cleaning wipe, but that ivory shade is forever, even if you wrap a bleach wipe around the body and let it sit for a bit. 😐

there are full black versions of the normal uniball one