Apple or Android? I was tripped up by Siri's volume until I figured out the independent volume control. That said, adjusting the volume while Siri was speaking worked for me (didn't change the music volume).
I haven’t had this problem (don't really use the feature), but have you tried changing the system volumes? It’s in General Settings under System. I haven't changed it from the factory, but looks like Text-to-Speech is pretty low by default.
In android auto I had to adjust the volume while it was "speaking" but it has been finicky at best, the volume settings is not a problem with the car but with android auto itself ... at least in my case, but even volume of 1 is plenty loud to me for Google