# How to collect the logs from multiple machines to my server efficiently? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 14, 2018, 10:35am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902 "2018-06-14T10:35:19Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Pawan\_Chandra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawan_chandra/32/31334_2.png) [@Pawan\_Chandra](https://discuss.elastic.co/u/Pawan_Chandra)\ **Post date:** [June 14, 2018, 10:35am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/1 "2018-06-14T10:35:19Z")
I want to send logs from multiple machines to my server via some end point. But I know that my files to be written is a log file which has the limits for a number of logs storage( maybe some 1000's). My logs would be anything around 10 million. What is the best possible scenario for log storage for me? Currently using filebeat to read the files from my local system, which is working fine. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 14, 2018, 2:39pm UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/2 "2018-06-14T14:39:35Z")
Hi @Pawan_Chandra, You are reading the logs with filebeat, but, where are you shipping them? I think that a good scenario is to collect logs with filebeat and ship them to an Elasticsearch cluster, you may also try to add Logstash between your filebeats and your Elasticsearch cluster depending on your needings. ---
**Author:** ![Pawan\_Chandra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawan_chandra/32/31334_2.png) [@Pawan\_Chandra](https://discuss.elastic.co/u/Pawan_Chandra)\ **Post date:** [June 15, 2018, 4:02am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/3 "2018-06-15T04:02:10Z")
Hello, @jsoriano obliged that you replied so quickly. the flow is: Log File--\>filebeat--\>logstash--\>1.elasticsearch\<--kibana |--------\>2. MongoDb My end to end flow is working completely fine. I have used my logstash config file with multiple outputs. One end is stdout, one is mongo and one is elasticsearch. My config file for logstash is : " input { beats { port =\> 5044 } } output { elasticsearch{ hosts =\> "localhost:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } mongodb{ uri =\> "mongodb://localhost:27017" database =\> "mycol" #name\_of\_database collection =\> "collab" #name\_of\_collection } stdout { codec =\> rubydebug } } " I just want to know what would be the most efficient way to store logs from multiple systems to mine. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 15, 2018, 9:54am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/4 "2018-06-15T09:54:09Z")
It seems that you have a good foundation to collect logs in an scalable way. The most efficient way depends on the systems you have, are you having any specific problem collecting logs with your current architecture? ---
**Author:** ![Pawan\_Chandra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawan_chandra/32/31334_2.png) [@Pawan\_Chandra](https://discuss.elastic.co/u/Pawan_Chandra)\ **Post date:** [June 15, 2018, 10:02am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/5 "2018-06-15T10:02:59Z")
As for now, I am trying to have multiple filebeat instances on different machines to collect their respective logs in my central machine. I am trying to figure it out that how that can happen because all I can see is: 1. Change the config output of the filebeat to my machine's IP address instead of regular "loalhost". 2. Start the filebeat from the remote machine. But its not working. Am I working in the right direction? If yes then how to move ahead? If NO or maybe are their any better alternatives? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 15, 2018, 10:06am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/6 "2018-06-15T10:06:09Z")
`filebeat` has to be installed in all machines you want to collect logs from, configured with the logstash output with the address of your logstash machine. ---
**Author:** ![Pawan\_Chandra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawan_chandra/32/31334_2.png) [@Pawan\_Chandra](https://discuss.elastic.co/u/Pawan_Chandra)\ **Post date:** [June 15, 2018, 10:09am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/7 "2018-06-15T10:09:01Z")
Thank you for your patience. Yes i have tried it but its now working. Is there any particular sequence of starting filebeat? or machines(central and remote) or shall i name them differently? filebeat 1or 2 etc. or is there anything to do with the registery file? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 15, 2018, 10:14am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/8 "2018-06-15T10:14:49Z")
You don't need to start filebeats in any particular sequence, but logstash should be running for the filebeats to be able to ship the logs. In logstash you need to configure the [input plugin for beats](https://www.elastic.co/guide/en/logstash/6.3/plugins-inputs-beats.html), and in beats the [logstash output](https://www.elastic.co/guide/en/beats/metricbeat/6.3/logstash-output.html). Once everything is configured, if it doesn't work you can check the logs of filebeats to see if they are reporting any problem. ---
**Author:** ![Pawan\_Chandra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawan_chandra/32/31334_2.png) [@Pawan\_Chandra](https://discuss.elastic.co/u/Pawan_Chandra)\ **Post date:** [June 15, 2018, 11:01am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/9 "2018-06-15T11:01:48Z")
thanks a lot. It worked. just one thing is there any limitation on the number of the filebeat instances? or the size limit for the log files in any system? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 15, 2018, 11:15am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/10 "2018-06-15T11:15:14Z")
In principle there is no limitation in the number of filebeat instances or size of stored logs, but you may need to fine-tune or scale your logstashs or the elasticsearch cluster to tolerate the load. You may also need to use multiple clusters if your collect a really big volume of logs. ---
**Author:** ![Pawan\_Chandra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pawan_chandra/32/31334_2.png) [@Pawan\_Chandra](https://discuss.elastic.co/u/Pawan_Chandra)\ **Post date:** [June 15, 2018, 11:22am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/11 "2018-06-15T11:22:16Z")
thanks a lot. Can you please throw some light over this? suppose , i will be having a roughly 10 million logs daily. and lets say i will be having 5 filebeat instances in different machines, where i am collecting my logs from multiple machines hitting an endpoint and distributing the logs equally on all 5 remote machines. what do you suggest(roughly) how many instances of elastic search or logstash or filebeat even I should be using for hassle free work . or what should be the ideal distribution? Thanks for your patience. so ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 15, 2018, 11:39am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/12 "2018-06-15T11:39:55Z")
It depends, you'd need to do some tests with your specific load. To give you an idea of the performance of Elasticsearch you can take a look to our [benchmarks](https://elasticsearch-benchmarks.elastic.co/index.html#tracks/http-logs/nightly/30d). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 13, 2018, 11:40am UTC](https://discuss.elastic.co/t/how-to-collect-the-logs-from-multiple-machines-to-my-server-efficiently/135902/13 "2018-07-13T11:40:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.