# Multisource index on elasticsearch passing by logstash **URL:** \ **Category:** Logstash\ **Created:** [April 26, 2023, 1:16pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844 "2023-04-26T13:16:23Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Abdeljalil\_El\_Yousso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdeljalil_el_yousso/32/120026_2.png) [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Post date:** [April 26, 2023, 1:16pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/1 "2023-04-26T13:16:23Z")
hey , im trying to create multiple source input from Filebeat , than injecting them into logstash to apply filters , and finally transfer the sources to elasticsearch as indexes The problem i have , only one index is created instaed of 2 in Elasticsearch , in which part i should specify my index name please ? ---
**Author:** ![sudhagar\_ramesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhagar_ramesh/32/105673_2.png) [@sudhagar\_ramesh](https://discuss.elastic.co/u/sudhagar_ramesh)\ **Post date:** [April 27, 2023, 4:13am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/2 "2023-04-27T04:13:31Z")
Hello @Abdeljalil_El_Yousso You need to create an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/indices-templates.html) and specify the number of shards to 1. This will create two index which means one primary and a replica. Index Templates | Elasticsearch Guide [2.4] | Elastic NOTE: Index template will be effective for new index and not for existing index. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 27, 2023, 4:40am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/3 "2023-04-27T04:40:10Z")
Hi @Abdeljalil_El_Yousso You need to share all you logstash pipeline confs? How are you defining them in pipelines.yml? ---
**Author:** ![Abdeljalil\_El\_Yousso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdeljalil_el_yousso/32/120026_2.png) [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Post date:** [April 27, 2023, 7:13am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/4 "2023-04-27T07:13:24Z")
```auto input { ##--------- put file path to analyse #file{ #path => " #start_position => "beginning" #sincedb_path => "nul" #} #stdin{} beats{ type => "filestream" port => 5044 } stdin{} } filter{ grok { match => {"message" => "\[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\] \[client %{IPORHOST:source_address}(:%{INT:source_port})\] %{DATA:[php][errorLevel]}\:%{GREEDYDATA:message} "} } #supprimer indesirable message #grok { #overwrite => "message" #} geoip { source => "source_address" ecs_compatibility => disabled target => "destination.geo" } mutate { add_field => { "locationn" => "%{geoip.location.lat},%{geoip.location.lon}" #rename => { "geoip.location.lat" => "[location][lon]" # "geoip.location.lon"=> "[location][lat]" } } } output{ #elasticsearch { # index => "apacheerror-%{+YYYY.MM.dd}" #index => "apacheapacheaccess-unity-%{+YYYY.MM.dd}" #hosts => ["https://localhost:9200"] #} if [type] == "filestream" { elasticsearch { hosts => ["https://localhost:9200"] user => " ********" password => " *****" index => "messages-%{+YYYY.MM.dd}" } #else elasticsearch { hosts => ["https://localhost:9200"] user => " ******" password => " ******" } } stdout{ } } ``` ---
**Author:** ![Abdeljalil\_El\_Yousso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdeljalil_el_yousso/32/120026_2.png) [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Post date:** [April 27, 2023, 7:14am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/5 "2023-04-27T07:14:01Z")
this is my configuration file on Logstash and im using Logsatsh 8.6 ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 27, 2023, 5:51pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/6 "2023-04-27T17:51:45Z")
Hi @Abdeljalil_El_Yousso Your config looks malformed. Here is a simple config that sends to 2 indices ```auto input { beats{ type => "filestream" port => 5044 } stdin{} } filter { } output { if [type] == "filestream" { elasticsearch { hosts => ["http://localhost:9200"] index => "my-index-%{+YYYY.MM.dd}" } } else { elasticsearch { hosts => ["http://localhost:9200"] index => "my-other-index-%{+YYYY.MM.dd}" } } stdout{ } } ``` ```auto GET /_cat/indices/my*?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open my-index-2023.04.27 b-zdx_d4Q-O0gvd2vAjYug 1 1 28998 0 11.3mb 11.3mb yellow open my-other-index-2023.04.27 s7AZzmezS7qZnEJTmbmVyA 1 1 3 0 13.8kb 13.8kb ``` ---
**Author:** ![Abdeljalil\_El\_Yousso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdeljalil_el_yousso/32/120026_2.png) [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Post date:** [April 28, 2023, 7:02am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/7 "2023-04-28T07:02:07Z")
what if i have 2 filestream in my configuration ? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 28, 2023, 7:08am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/8 "2023-04-28T07:08:26Z")
> [@Abdeljalil\_El\_Yousso](#): > > what if i have 2 filestream in my configuration ? Not sure I understand the question.... Do you mean file streams coming from filebeat or file streams coming from logstash file input?.. In the end you just need a tag them and then do if else on the outputs. You can tag inputs and filebeat, then use those in logstash It's programming.... Set fields the use those fields in conditions ---
**Author:** ![Abdeljalil\_El\_Yousso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdeljalil_el_yousso/32/120026_2.png) [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Post date:** [April 28, 2023, 7:23am UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/9 "2023-04-28T07:23:31Z")
Thanks for your quick reply. What i want to say , if you have 2 filestream inputs in Filebeats , and your output is Logstash , on your logstash configuration file , you specity the output to Elasticsearch . In this case , how can you create 2 indexes ? hope my question is clear AND THANKS AGAIN ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 28, 2023, 3:34pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/10 "2023-04-28T15:34:27Z")
Roll Up your sleeves and read the docs... [here](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) is one way. there are multiple ways... Here is one... Tag each filebeat input ```auto - type: filestream # Unique ID among all inputs, an ID is required. id: my-filestream-id enabled: true paths: - /var/log/*.log # Add a tag to be used later tags: ["app-type-1"] ... ``` Then logstash output tags are an array so it looks a little different ```auto output { if "app-type-1" in [tags]{ elasticsearch { hosts => ["http://localhost:9200"] index => "my-type-1-index-%{+YYYY.MM.dd}" } } else if "app-type-2" in [tags] { elasticsearch { hosts => ["http://localhost:9200"] index => "my-type-2-index-%{+YYYY.MM.dd}" } } else { elasticsearch { hosts => ["http://localhost:9200"] index => "my-type-other-index-%{+YYYY.MM.dd}" } } stdout{ } } ``` Good Luck! Dig In! ---
**Author:** ![Abdeljalil\_El\_Yousso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdeljalil_el_yousso/32/120026_2.png) [@Abdeljalil\_El\_Yousso](https://discuss.elastic.co/u/Abdeljalil_El_Yousso)\ **Post date:** [April 28, 2023, 3:57pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/11 "2023-04-28T15:57:42Z")
Thank you so much. I managed to create multiple index after ur 2nd reply by adding Tags in both Filebeats and Logstach. Thanks again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 26, 2023, 3:58pm UTC](https://discuss.elastic.co/t/multisource-index-on-elasticsearch-passing-by-logstash/330844/12 "2023-05-26T15:58:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.