Ok If anyone else wishes to do the same as I have, here is how it works.
You need an interface between the new headunit and the factory amp. This unit is called a METRA TYTO-01 2003-08 TOYOTA JBL INTERFACE WIRE HARNESS. It plugs directly into your original plug, and sends the correct data down the TX+ and TX- wires, and also has a gain adjustment for varying your rca signal up or down to get the right level for your amp.
If you are going to connect other functions to the head unit, such as a dimmer for lights on, reverse camera etc, you will still need to splice wires into the harness.
If you use this interface you have no need to bypass the factory amp, or replace the rear speakers in order to install your aftermarket head unit. (you can have your cake and eat it).
The installation I have done gives just as superb sound quality from my aftermarket head unit as the original head unit, with all the extra functions you want these days in a Lexus.