Update:
Decided to ring up Brighton again. Ended up speaking to a different person in the Service Department. This time I was quoted $780.
The Toyota Dealer quoted me $670. They explained (and as others have in this forum) that there is quite a bit of work involved with changing the fuel filter with having to remove the seats, etc.
Not much difference (relatively) between the Toyota and Brighton quote, so decided to book it in with Brighton.