One of the most important challenges of building an ELK deployment is making it scalable. Logstash may fail when attempting to index logs in Elasticsearch that cannot match into the automatically-generated mapping. It is very susceptible to load, which implies you should be extremely careful when indexing and rising your amount of documents. When Elasticsearch is busy, Logstash works slower than normal — which is the place your buffer comes into the picture, accumulating extra documents that can then be pushed to Elasticsearch. The really helpful method to ensure a resilient knowledge pipeline is to put a buffer in front of Logstash to act as the entry level for all log events which would possibly be shipped to your system. It will then buffer the data till the downstream elements have enough sources to index.
