# Logstash - ES index mixed up **URL:** \ **Category:** Logstash\ **Created:** [March 20, 2024, 9:27am UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796 "2024-03-20T09:27:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![maskrider1111](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maskrider1111/32/116613_2.png) [@maskrider1111](https://discuss.elastic.co/u/maskrider1111)\ **Post date:** [March 20, 2024, 9:27am UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/1 "2024-03-20T09:27:33Z")
Hey folks, Currently im trying to ingest multiple firewall log source by differencing the folders and index. I created conf file for each source so that it will be ingested separately into different index. However i stumbled into an issue where all the log source are mixed up with the index. In short, all the index contain data from different log source although i separated it in conf level. Can someone point out what would be my mistake here? Following is on the the conf content: ```auto input { file { path => "/logstash/fortigate_anomaly_log/*.log" start_position => "beginning" sincedb_path => "/var/lib/logstash/sincedb" } } filter { grok { match => ["message", "%{NOTSPACE:devname} %{NOTSPACE:dev ice_id} %{IPORHOST:rempip} %{IPORHOST:locip} %{NOTSPACE:msg} %{NOTSPACE:group}"] } kv {} mutate { add_field => { "eventtime" => 1675199470000000000 } } mutate { gsub => ["eventtime", "\d{6}$", ""] } date { match => ["[eventtime][0]","UNIX_MS","ISO8601" ] target => "@timestamp" timezone => "UTC" } mutate { convert => { "sentbyte" => "integer" } convert => { "rcvdbyte" => "integer" } } } output { elasticsearch { hosts => "localhost" index => "fortigate-anomalylog" } stdout{ codec => rubydebug } } ``` ```auto path => "/logstash/fortigate_anomaly_log/*.log" ``` The path is changed for diff log source. Following is the index screenshot which appear to be mixed up and contain same data across the index: ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/3/63e57ed000713faf26b6a392f5ab7e857cf9c38c.png) ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [March 20, 2024, 9:50am UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/2 "2024-03-20T09:50:25Z")
> [@maskrider1111](#): > > contain data from different log source although i separated it in conf level. You have 6 logs which are a different internal structure. You are reading everything from the same directory?, and written in fortigate-anomalylog index. Cannot get logic why you haven't use IFs and fields from logs to make distinguished from each other also is the output part. Also you can make 6 pipelines and process separately. ---
**Author:** ![maskrider1111](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maskrider1111/32/116613_2.png) [@maskrider1111](https://discuss.elastic.co/u/maskrider1111)\ **Post date:** [March 20, 2024, 10:19am UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/3 "2024-03-20T10:19:45Z")
Hi Rios, Thanks for the reply. First, im trying out the ELK stack to ingest the firewall log and honestly I'm not an expert. This is something that i came out by going through blogs and try-error myself. If there is optimization that could be done, sure I will but for now this is merely on testing phase and i really appreciate your comments. To answer your question, im reading the logs from different sub-directory and each having its own config file. ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00f9edc4f3dd6ffb92cc7957aa858cdc2496eaa7.png) ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [March 20, 2024, 12:22pm UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/4 "2024-03-20T12:22:04Z")
You must know the field structure to parse correctly. The KV filter is useful however in some cases logs can have sort of header which need to be parsed by grok or dissect. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 20, 2024, 12:58pm UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/5 "2024-03-20T12:58:04Z")
> [@maskrider1111](#): > > To answer your question, im reading the logs from different sub-directory and each having its own config file How are you running logstash? As a systemd service? If you have multiple configurations and want them to be executed as separated pipelines, you need to configura logstash to run multiple pipelines as explained in the [documentation](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html). Per default logstash will run everything inside `/etc/logstash/conf.d` as a single pipeline, it will merge all files inside this folder in one pipeline and all data received by the input will pass through all the filters and be sent to all the outputs, unless you use conditionals to filter this. ---
**Author:** ![maskrider1111](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maskrider1111/32/116613_2.png) [@maskrider1111](https://discuss.elastic.co/u/maskrider1111)\ **Post date:** [March 20, 2024, 2:45pm UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/6 "2024-03-20T14:45:05Z")
Hi @leandrojmp, That make sense (definitely) and now i changed the pipeline.yml config. The new issue is the defined pipelines not loaded after i restart the logstash service ```auto curl -XGET "localhost:9600/_node/pipelines?pretty" "pipelines" : { "main" : { "ephemeral_id" : "redacted", "hash" : "redacted", "workers" : 12, "batch_size" : 125, "batch_delay" : 50, "config_reload_automatic" : false, "config_reload_interval" : 3000000000, "dead_letter_queue_enabled" : false } ``` I even change the the configuration path in logstash.yml (although its not needed, according to chatgpt). Unfortunately no luck and I'm still seeing the main pipeline loaded. Did i miss something here? ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/8/88132c58fa68d980c08161178a7357fa5ecf9595.png) ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [March 20, 2024, 4:15pm UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/7 "2024-03-20T16:15:44Z")
You are not receiving data if you are using "old" files, recoded in sincedb\_path. Try only with one pipeline, set: `sincedb_path => "/dev/null"` If data come to ES, a temp index names, that means your files were processed. Not sure is the filter section OK or not, since we don't have data, yet. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 20, 2024, 4:49pm UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/8 "2024-03-20T16:49:16Z")
> [@maskrider1111](#): > > Did i miss something here? You didn't said how you are running logstash. Are you running it as a service? Also, what do you have in the logs after you restarted it? Please share the logs. There is no `main` pipeline in your `pipelines.yml`, so your logstash is not using it, this can happen if you are not running as a service or are using the `-f` parameter when using the command line. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 17, 2024, 4:49pm UTC](https://discuss.elastic.co/t/logstash-es-index-mixed-up/355796/9 "2024-04-17T16:49:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.