# Filebeat to logstash multiple indexs **URL:** \ **Category:** Logstash\ **Created:** [November 28, 2016, 8:41pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399 "2016-11-28T20:41:05Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 28, 2016, 8:41pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/1 "2016-11-28T20:41:05Z")
i'm new to ELK was trying to configure filebeat on multiple instances. i have different types of logs. 1. kafka logs 2. zookeeper logs 3. hdfs logs 4. yarn logs . . . on all these instances i was trying to configure file beat and from around 50 filebeats i was sending logs to single logstash. So i was wondering how can i differentiate all this logs. do it works with multple index names or filers on logstash or something else?? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 28, 2016, 9:44pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/2 "2016-11-28T21:44:23Z")
> So i was wondering how can i differentiate all this logs. You can use conditionals in the Logstash configuration in order to do different things for different kinds of events. > do it works with multple index names or filers on logstash or something else?? You can send different kinds of events to different indexes, yes. Your question is very open so it's hard to be specific. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 29, 2016, 3:00am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/3 "2016-11-29T03:00:03Z")
> [@magnusbaeck](#): > > You can send different kinds of events to different indexes, yes. if possible can you please send basic syntax for different indexs with single logstash with multiple filebeat ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 29, 2016, 7:15am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/4 "2016-11-29T07:15:23Z")
> ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 29, 2016, 1:57pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/5 "2016-11-29T13:57:28Z")
I was sending logs from multiple filebeats to single logstash and then to elastic search. filebeats(count 20) ---\> logstash(1) --\> elasticsearch(1) ---\> kibana does it works the example you send to me was directly from filebeat to elastic search. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 29, 2016, 2:18pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/6 "2016-11-29T14:18:35Z")
> does it works the example you send to me was directly from filebeat to Elasticsearch. The example I gave had nothing at all to do with Filebeat. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 29, 2016, 2:31pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/7 "2016-11-29T14:31:50Z")
ok got you . filebeat just push the data , logstash will do all filter. thanks. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 29, 2016, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/8 "2016-11-29T14:35:32Z")
can i get official repo for logstash and filebeat. I was trying to configure logstash on my instance i was getting facing issue. i dont know if i was doing anything wrong I tried on both ubuntu & centos i was facing same issue with 5.0.1 but 2.4 works for me.I wan't to use latest version. This repo i was trying to use. [https://www.elastic.co/guide/en/logstash/current/installing-logstash.html](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html) ## **[root@ip-**\*\*\*\*\*\*]# service logstash status\*\* **logstash: unrecognized service** I was doing these setps to configure logstash rpm --import [https://artifacts.elastic.co/GPG-KEY-elasticsearch](https://artifacts.elastic.co/GPG-KEY-elasticsearch) Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo [logstash-5.x] name=Elastic repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md And your repository is ready for use. You can install it with: ## sudo yum install logstash ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 30, 2016, 12:06am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/9 "2016-11-30T00:06:50Z")
starting agent {:level=\>:info, :file=\>"logstash/agent.rb", :line=\>"207", :method=\>"execute"} starting pipeline {:id=\>"main", :level=\>:info, :file=\>"logstash/agent.rb", :line=\>"469", :method=\>"start\_pipeline"} Settings: Default pipeline workers: 1 Beats inputs: Starting input listener {:address=\>"0.0.0.0:5044", :level=\>:info, :file=\>"logstash/inputs/beats.rb", :line=\>"111", :method=\>"register"} Beats inputs: Starting input listener {:address=\>"0.0.0.0:5044", :level=\>:info, :file=\>"logstash/inputs/beats.rb", :line=\>"111", :method=\>"register"} Pipeline aborted due to error {:exception=\>#\, :backtrace=\>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:853:in`new'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/lumberjack/beats/server.rb:51:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.9/lib/logstash/inputs/beats.rb:119:in`register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:330:in `start_inputs'", "org/jruby/RubyArray.java:1613:in`each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:329:in `start_inputs'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:180:in`start\_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:in`start\_pipeline'"], :level=\>:error, :file=\>"logstash/agent.rb", :line=\>"475", :method=\>"start\_pipeline"} stopping pipeline {:id=\>"main", :file=\>"logstash/agent.rb", :line=\>"388", :method=\>"shutdown\_pipelines"} Can you pleas help me with this error !!! Thanks. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 30, 2016, 6:28am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/10 "2016-11-30T06:28:50Z")
It looks like you've defined to beats inputs, both trying to listen on the same port. Keep in mind that Logstash reads _all_ files in /etc/logstash/conf.d. Perhaps you have a left-over backup file or similar? ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 30, 2016, 3:02pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/11 "2016-11-30T15:02:28Z")
My question was, I know i can send 10 kafka logs using beats to single logstash, but my question was, if i have for example 10 kafka (beats) , 5 zookeepers(beats), 4 spark(beats) ...... can i send all different types to single logstash with different indexs ?? input { beats { type =\> "kafka" port =\> "5044" } beats { type =\> "zookeeper" port =\> "5044" } } output { if [type] == "kafka" { elasticsearch { action =\> "index" hosts =\> "elasticsearchip:80" index =\> "kafkalogs" } stdout { codec =\> rubydebug } } else { elasticsearch { action =\> "index" hosts =\> "elasticsearchip:80" index =\> "zookeeperlogs" } stdout { codec =\> rubydebug } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 30, 2016, 3:07pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/12 "2016-11-30T15:07:08Z")
> can i send all different types to single logstash with different indexs ?? Yes, but you obviously can't have multiple beats listeners using the same port. Either use multiple ports or use a single listener and use some other method to distinguish between different kind of events. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 30, 2016, 3:26pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/13 "2016-11-30T15:26:05Z")
if possible can you please provide me sample syntax !! can you please tell me some other ports than 5044, i will try those. [quote="magnusbaeck, post:12, topic:67399"] Yes, but you obviously can't have multiple beats listeners using the same port. Either use multiple ports or use a single listener and use some other method to distinguish between different kind of events. [/quote] ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 30, 2016, 3:27pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/14 "2016-11-30T15:27:46Z")
> if possible can you please provide me sample syntax !! An example of what? > can you please tell me some other ports than 5044, i will try those. You can use any port that's available on your machine. Try 5045, for example. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [November 30, 2016, 8:10pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/15 "2016-11-30T20:10:01Z")
> [@magnusbaeck](#): > > An example of what? i was not sure where i was wrong, was creating index only for zookeeper logs but not for kafka logs. can you please correct syntax if i was doing anything wrong input { beats { type =\> "kafka" port =\> "5044" } beats { type =\> "zookeeper" port =\> "5045" } } output { if [type] == "kafka" { elasticsearch { action =\> "index" hosts =\> "elasticsearchip:80" index =\> "kafkalogs" } stdout { codec =\> rubydebug } } else { elasticsearch { action =\> "index" hosts =\> "elasticsearchip:80" index =\> "zookeeperlogs" } stdout { codec =\> rubydebug } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 2, 2016, 6:52am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/16 "2016-12-02T06:52:50Z")
That looks correct. The Logstash logs should contain more clues about what's going on. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [December 2, 2016, 4:10pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/17 "2016-12-02T16:10:30Z")
i was not sure where it was getting wrong, this error i was getting Reading config file {:config\_file=\>"/etc/logstash/conf.d/logstash.conf", :level=\>:debug, :file=\>"logstash/config/loader.rb", :line=\>"69", :method=\>"local\_config"} fetched an invalid config {:config=\>"input {\n beats {\n port =\> 5044\n tags =\> ["bh-test"]\n }\nfilter{\nif [type] == "clouda" {\n grok {\n match =\> [{ "message" =\> "%{cloud-init}" }]\n }\n }\n\n}\noutput {\nif "bh-test" in [tags] {\n elasticsearch {\n action =\> "index"\n hosts =\> "endpoint:80"\n index =\> "w-test"\n }\n }\n}\n\n", :reason=\>"Expected one of #, =\> at line 7, column 4 (byte 82) after input {\n beats {\n port =\> 5044\n tags =\> ["bh-test"]\n }\nfilter{\nif ", :level=\>:error, :file=\>"logstash/agent.rb", :line=\>"430", :method=\>"create\_pipeline"} starting agent {:level=\>:info, :file=\>"logstash/agent.rb", :line=\>"207", :method=\>"execute"} filebeat:: - input\_type: log paths: - /var/log/cloud-init.log document\_type: clouda tags: ["bhtest"] logstash:: input { beats { port =\> 5044 tags =\> ["bhtest"] } filter{ if [type] == "clouda" { grok { match =\> [{ "message" =\> "%{cloud-init}" }] } } } output { if "bhtest" in [tags] { elasticsearch { action =\> "index" hosts =\> "ip:80" index =\> "w-test" } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 4, 2016, 10:28am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/18 "2016-12-04T10:28:50Z")
> ``` > match => [{ "message" => "%{cloud-init}" }] > > ``` Change to: ``` match => { "message" => "%{cloud-init}" } ``` This might not be what Logstash is complaining about. I can't spot what it otherwise could be though. Comment out blocks to narrow things down and consider running the config file through e.g. hexdump to make sure you don't have any invisible garbage characters. ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [December 4, 2016, 11:57pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/19 "2016-12-04T23:57:34Z")
I was trying something like this, but was not able to see any logs & index in kibana. my main problem was trying to get different indexs for different files i have please help me with this issue. filebeat: prospectors: - paths: - /var/log/redis/\*.log document\_type: redis ``` - paths: - /var/log/python/*.log document_type: python - paths: - /var/log/mongodb/*.log document_type: mongodb ``` input { beats { port =\> 5044 } } output { # Customize elasticsearch output for Filebeat. if [@metadata][beat] == "filebeat" { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false # Use the Filebeat document\_type value for the Elasticsearch index name. index =\> "%{[@metadata][type]}-%{+YYYY.MM.dd}" document\_type =\> "log" } } } ---
**Author:** ![bob-bza](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Post date:** [December 5, 2016, 1:08am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399/20 "2016-12-05T01:08:24Z")
> [@Multiple document- types in filebeat](https://discuss.elastic.co/t/multiple-document-types-in-filebeat/44897/2): > > The formatting of your configuration is mangled so I'm not sure exactly what it looks like (always post Filebeat configuration formatted as code), but it seems to be this: filebeat: prospectors: - paths: - "/home/syslog/filebeat/redware/\*.log" document\_type: syslog paths: - "/home/voip/cdr\_StandAloneCluster\_01\_2016\*" document\_type: cucm-cdr If that's the case then you're missing a "-" before the second "paths" declaration. See the example at [http…](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html) i was looking for this , but forward this to logstash and from there to elasticsearch with multiple index [Next page](https://discuss.elastic.co/t/filebeat-to-logstash-multiple-indexs/67399.md?page=2)