Also, it depends on the DAW you are using, many DAWs (Logic, Cubase, S1β¦) have so called βhybridβ engines, where tracks that are not record armed are played through a much higher βsafety bufferβ anyway and the buffer settings of the audio interface isnβt as relevant.
Also, with higher buffer sizes, it is advisable to have an eye on automation, as it a) can lose precision and b) actually produce nonsense (Iβve had that with specific Cubase versions where a buffer size of higher than 2048 would produce some sorts of βhallucinationsβ in the render. A bug of course).