The buffer size setting is one that will confuse the newbie audio engineer. It determines the rate at which data is sent and received to and from the audio interface. The smaller the latency, the faster the information is sent.
So, why not always keep it at the fastest setting? Data drop outs. Clicks, pops, crackle and sometimes software crashes. Here are a few videos on the topic of buffer sizes:
The key points to remember:
When recording, use the lowest buffer size as possible. When mixing, change it to a higher setting like 1024 or 2048.
The standard buffer sizes are 32, 64, 128, 256, 512, 1024, 2048, 4096 and 8192.
Higher sample rates = lower latency. 128 samples at 44.1 kHz is slower than 128 samples at 96 kHz, for example.
Drivers and how your computer is set up matters. In general, recording with a desktop is preferable to a laptop. PCI/PCIe and Thunderbolt (essentially, external PCIe) interfaces are better than USB.
Speaking of USB, I recently had severe problems after upgrading my USB host controller/hub drivers. Downgrading to the old version fixed that issue.
RME is the best company out there in terms of latency and driver stability. Their interfaces can handle larger amounts of CPU usage than other companies’ interfaces. Latency is always a hot topic of discussion, which explains the 50+ pages on this Gearslutz thread. But, with the information from the above videos you will know all that you need to know.