Your understanding is correct. But I can think of a way to streamline your case a bit.
Each buffer still needs to go in a separately named section. Presume the sections are named section_for_buffer_one, section_for_buffer_two, and so on. In the link command file, you can combine them into a single output section that is allocated to a hard-coded address. Something like ...
output_section_for_buffers {
*(section_for_buffer_one)
*(section_for_buffer_two)
...
} run = 0x320, type = NOINIT
As before, use NOINIT only on the DSP side. The sections/buffers are placed in the output section in that order.
Another improvement I should have mentioned earlier. The linker supports the simpler parts of C preprocessing, such as #include and #define. So, you can define names for the hard-coded addresses in one file that is included by the ARM and DSP link command files. Then use those names where you presently have the hard-coded numbers.
Thanks and regards,
-George