I can't pinpoint what exactly is going on. But, if it was my car with that problem and those amount of miles I would consider my rack to be the problem.
But, I can't say it isn't your CV joints, it possibly could be. Worn out CV joints are usually accompanied with a clicking noise while turning.
Regardless of if the problem is the CV joints or the rack, you should change out that PS fluid. Nasty fluid can have adverse effects on the system, and that fluid could possibly be original. To do this you need some type F ATF just some regular kind, not sythetic, and a baster. Simply, use the baster to suck out the PS resivour and then refill it with the new fluid. Then drive the car for a day or two and do it again. Keep doing this for a week to two weeks untill the fluid gets clean and clear and stays that way.
I would suggest this way over just draining the whole system and refilling it because it slowly introduces the new fluid into the system and IMHO is a cleaner, easier, way to do it. And you arn't stuck needing to bleed the system afterwards.
Doug