# Index not displaying expected log values

**URL:** <https://discuss.elastic.co/t/index-not-displaying-expected-log-values/282314>\
**Category:** Elasticsearch\
**Created:** [August 24, 2021, 8:15am UTC](https://discuss.elastic.co/t/index-not-displaying-expected-log-values/282314 "2021-08-24T08:15:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![flenoir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flenoir/32/93623_2.png) [@flenoir](https://discuss.elastic.co/u/flenoir)\
**Post date:** [August 24, 2021, 8:15am UTC](https://discuss.elastic.co/t/index-not-displaying-expected-log-values/282314/1 "2021-08-24T08:15:56Z")

</div>

Hi,

I'm struggling with filebeat and logstash because, i want to handle different logs ( logs and json file) as inputs and i want to have a dedicated index for all logs (first\_index and another one for json datas ( on second\_index). but as a result, first\_index is ok but , on my second\_index i have all logs datas where i just expected amneo-connector.ndjson datas.

My filebeat config is

```auto
filebeat.inputs:
- type: log
 paths:
    - /var/log/apache2/access-toto-magento.log
  fields:
    file_name: access-toto-magento.log
    log_type: apache
    environment: in01

- type: log
  paths:
    - /var/log/apache2/error-toto-magento.log
  fields:
    file_name: error-toto-magento.log
    log_type: apache
    environment: in01
  
- type: log
  paths:
    - /var/www/toto-magento/current/var/log/system.log
  fields:
    file_name: system.log
    log_type: magento
    environment: in01

- type: log
  paths:
    - /var/www/toto-magento/current/var/log/amneo-connector.ndjson
  fields:
    file_name: amneo-connector.ndjson
    log_type: json
    environment: in01
  json.keys_under_root: true
  json.add_error_key: true

processors:
 - decode_json_fields:
     fields: ["message"]
  
#----------------------------- Logstash output --------------------------------

output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # The Logstash hosts
  hosts: ["amneo-corenonprod-logstash.amneo-corenonprod.int:5044"]

logging.level: warning

logging.to_files: true

# Send all logging output to syslog. The default is false.
logging.to_syslog: false

logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7
  permissions: 0644

```

and my logstash config is

```auto

input {
    beats {
        port => "5044"
        ssl => false
    }
}
filter {
    if "access-toto-magento.log" in [fields][file_name] {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
        geoip {
            source => "clientip"
       }
        mutate {
          remove_field => ["beat"]
        }
        mutate {
          add_tag => ["logs"]
        }
    }
    if "error-toto-magento.log" in [fields][file_name] {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => { "message" => "%{APACHE_ERROR_LOG}" }
        }
        mutate {
          remove_field => ["beat"]
        }
        mutate {
          add_tag => ["logs"]
        }
    }
     if "system.log" in [fields][file_name] {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => { "message" => "%{MAGENTO_LOGS}" }
        }
        mutate {
          remove_field => ["beat"]
        }
        mutate {
          add_tag => ["logs"]
        }
    }
    if "tesb.log" in [fields][file_name] {
        grok {
           patterns_dir => "/etc/logstash/patterns"
           match => { "message" => "%{TALEND_TESB_LOGS}" }
       }
        mutate {
          remove_field => ["beat"]
        }
        mutate {
          add_tag => ["logs"]
        }
    }
    if "toto.log" in [fields][file_name] {
        grok {
           patterns_dir => "/etc/logstash/patterns"
           match => { "message" => "%{TALEND_TOTO_LOGS}" }
       }
        mutate {
          remove_field => ["beat"]
        }
        mutate {
          add_tag => ["logs"]
        }
    }
     if "amneo-connector.ndjson" in [fields][file_name] {
        grok {
            match => { "message" => "%{GREEDYDATA:json_data}" }
        }
        mutate {
          add_tag => ["ndjson"]
        }
    }
}
output {
    if "logs" in [tags] {
        amazon_es {
            hosts => "search-amneo-corenonprod-1-hr2touhg2mcgdis4f6y4jvktli.eu-west-1.es.amazonaws.com"
            region => "eu-west-1"
            index => "first_index-%{+YYYY.MM.dd}"
            #max_bulk_bytes => 9999999
        }
    }
    if "ndjson" in [tags] {
         amazon_es {
            hosts => "search-amneo-corenonprod-1-hr2touhg2mcgdis4f6y4jvktli.eu-west-1.es.amazonaws.com"
            region => "eu-west-1"
            index => "second_index-%{+YYYY.MM.dd}"
            #max_bulk_bytes => 9999999
        }
    }
}

```

can you help understand why my logs datas are appearing in second\_index ??

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 25, 2021, 1:20am UTC](https://discuss.elastic.co/t/index-not-displaying-expected-log-values/282314/2 "2021-08-25T01:20:46Z")

</div>

Have you tried using [`fields`](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-fields) directly in Filebeat?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 22, 2021, 1:21am UTC](https://discuss.elastic.co/t/index-not-displaying-expected-log-values/282314/3 "2021-09-22T01:21:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.