# When outputting from logstash to elasticsearch, some data is missing

**URL:** <https://discuss.elastic.co/t/when-outputting-from-logstash-to-elasticsearch-some-data-is-missing/307820>\
**Category:** Logstash\
**Created:** [June 22, 2022, 2:03am UTC](https://discuss.elastic.co/t/when-outputting-from-logstash-to-elasticsearch-some-data-is-missing/307820 "2022-06-22T02:03:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sbyunnn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbyunnn/32/107387_2.png) [@sbyunnn](https://discuss.elastic.co/u/sbyunnn)\
**Post date:** [June 22, 2022, 2:03am UTC](https://discuss.elastic.co/t/when-outputting-from-logstash-to-elasticsearch-some-data-is-missing/307820/1 "2022-06-22T02:03:41Z")

</div>

Hi, I am building ELK Stack  
The output of my logstash is Elasticsearch  
but, some data is missing in ES

However, some data is missing from ES.  
After removing all indexes, I can see missed data. but, the other data missing

I checked logstash's stdout, and nothing is missing

Why is the data missing?  
help me T.T

There are a total of 6 nodes.  
3 are master and 3 are data nodes.

I set logstash to 3 data nodes

my pipeline is

```auto
input {
  kafka {
    bootstrap_servers => "192.168.xxx.xxx:9092,192.168.xxx.xxx:9092,192.168.xxx.xxx:9092"
    group_id => "dev-all"
    topics => ["dev-all"]
    consumer_threads => 1
    decorate_events => true
    }
}

filter {
 json {
   source => "message"
 }
}

output{
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["192.168.xxx.xxx:9200","192.168.xxx.xxx:9200","192.168.xxx.xxx:9200"]
    index => "dev-all-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "${es_password}"
  }
}

```

below is logstash's stdout ( Omit some unnecessary data)

```auto
{
       "message" => "....",
          "time" => "2022-06-22T01:51:20.547705299Z",
      "@version" => "1",
    "kubernetes" => {
              ...
               "pod_name" => "ingress-nginx-controller-lwnsr",
         "namespace_name" => "ingress-nginx",
         "container_name" => "controller"
    },
        "stream" => "stdout",
    "@timestamp" => 2022-06-22T01:51:20.547Z,
           "log" => "..."
}		   
{
       "message" => "...",
          "time" => "2022-06-22T01:50:26.597591464Z",
      "@version" => "1",
    "kubernetes" => {
              ...
               "pod_name" => "logtest-58b9747db5-xcqbz",
         "namespace_name" => "logging-test",
         "container_name" => "count"
    },
        "stream" => "stdout",
    "@timestamp" => 2022-06-22T01:50:26.597Z,
           "log" => "..."
}

```

There are two data, "namespace\_name" key is 'ingress-nginx' and 'logging-test' data.

I can see 'ingress-nginx' data

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/b/6b7171fee0fe7731969fb0c985fe7c4080c28c88.png)

but I can not see 'logging-test' data

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/1/8119f27d84e7d5a6ff1ec5ee33bb6c03b634ab7f.png)

please help me..T.T

---

<div class="post-metadata">

**Author:** ![sbyunnn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbyunnn/32/107387_2.png) [@sbyunnn](https://discuss.elastic.co/u/sbyunnn)\
**Post date:** [June 23, 2022, 1:00am UTC](https://discuss.elastic.co/t/when-outputting-from-logstash-to-elasticsearch-some-data-is-missing/307820/2 "2022-06-23T01:00:13Z")

</div>

When the UTC based date changes, as a new index is created, I checked that there is no 'ingress-nginx' data and the 'logging-test' data is visible.  
Why? Let me know what I did wrong...

---

<div class="post-metadata">

**Author:** ![sbyunnn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sbyunnn/32/107387_2.png) [@sbyunnn](https://discuss.elastic.co/u/sbyunnn)\
**Post date:** [June 23, 2022, 4:59am UTC](https://discuss.elastic.co/t/when-outputting-from-logstash-to-elasticsearch-some-data-is-missing/307820/3 "2022-06-23T04:59:21Z")

</div>

I found the cause.  
Because the index template was not specified, it tried to save in a different format.  
So when the newly index create, data was missing.  
I set different format of index template as 'flattened'  
Thanks you 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2022, 4:59am UTC](https://discuss.elastic.co/t/when-outputting-from-logstash-to-elasticsearch-some-data-is-missing/307820/4 "2022-07-21T04:59:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.