# Data not getting indexed in elastic search

**URL:** <https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207>\
**Category:** Logstash\
**Created:** [April 16, 2018, 2:22pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207 "2018-04-16T14:22:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [April 16, 2018, 2:22pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/1 "2018-04-16T14:22:10Z")

</div>

[root@ukmr0xtmd01 conf.d]# /usr/share/logstash/bin/logstash -f siae.conf  
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults  
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console  
[INFO] 2018-04-16 15:19:35.812 [main] scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}  
[INFO] 2018-04-16 15:19:35.818 [main] scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}  
[WARN] 2018-04-16 15:19:36.333 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[INFO] 2018-04-16 15:19:36.530 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"6.2.3"}  
[INFO] 2018-04-16 15:19:36.809 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9601}  
[INFO] 2018-04-16 15:19:38.093 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[INFO] 2018-04-16 15:19:38.356 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://127.0.0.1:9200/](http://127.0.0.1:9200/)]}}  
[INFO] 2018-04-16 15:19:38.360 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://127.0.0.1:9200/](http://127.0.0.1:9200/), :path=\>"/"}  
[WARN] 2018-04-16 15:19:38.449 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=\>"[http://127.0.0.1:9200/](http://127.0.0.1:9200/)"}  
[INFO] 2018-04-16 15:19:38.632 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es\_version=\>6}  
[WARN] 2018-04-16 15:19:38.636 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[INFO] 2018-04-16 15:19:38.644 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=\>nil}  
[INFO] 2018-04-16 15:19:38.647 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[INFO] 2018-04-16 15:19:38.672 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//127.0.0.1](https://127.0.0.1)"]}  
[INFO] 2018-04-16 15:19:38.978 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started succesfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x2c2e84dc@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run\>"}  
[INFO] 2018-04-16 15:19:39.007 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=\>1, :pipelines=\>["main"]}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2018, 2:30pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/2 "2018-04-16T14:30:52Z")

</div>

OK, so it started normally. What does the configuration look like? If you add a stdout output do you see any events?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 16, 2018, 2:30pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/3 "2018-04-16T14:30:52Z")

</div>

And what's in siae.conf?

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [April 16, 2018, 2:40pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/4 "2018-04-16T14:40:37Z")

</div>

this is the name if the config file.

input{  
file {  
path =\> "/data/input/TXN\_NEMs/SIAE\_NSM5UX/\*\*/\*"  
start\_position =\> "beginning"  
type =\> "siae"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
if [type] == "siae" {  
grok {  
patterns\_dir =\> ["/etc/logstash/conf.d/patterns"]  
match =\> { "message" =\> "%{CISCOTIMESTAMP:update\_time} %{WORD:name} %{NOTSPACE:id}: %{GREEDYDATA:msg}" }

```
  }
  if ("sshd" in [id]) {
            mutate { add_tag => "shouldexist" }
  }

 date {
    match => ["update_time" , "MMM dd HH:mm:ss", "MMM d HH:mm:ss"]
    locale => "en"
 }

if "_grokparsefailure" in [tags] {
    drop {}
}

```

}  
}

output {  
if [type] == "siae" {  
if "shouldexist" in [tags] {  
elasticsearch {  
hosts =\> ["127.0.0.1"]  
index =\> "siae-%{+YYYY.MM.dd}"  
}

# stdout { codec =\> rubydebug }

}  
}  
}

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [April 16, 2018, 2:41pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/5 "2018-04-16T14:41:02Z")

</div>

no  
input{  
file {  
path =\> "/data/input/TXN\_NEMs/SIAE\_NSM5UX/\*\*/\*"  
start\_position =\> "beginning"  
type =\> "siae"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
if [type] == "siae" {  
grok {  
patterns\_dir =\> ["/etc/logstash/conf.d/patterns"]  
match =\> { "message" =\> "%{CISCOTIMESTAMP:update\_time} %{WORD:name} %{NOTSPACE:id}: %{GREEDYDATA:msg}" }

```
  }
  if ("sshd" in [id]) {
            mutate { add_tag => "shouldexist" }
  }

 date {
    match => ["update_time" , "MMM dd HH:mm:ss", "MMM d HH:mm:ss"]
    locale => "en"
 }

if "_grokparsefailure" in [tags] {
    drop {}
}

```

}  
}

output {  
if [type] == "siae" {  
if "shouldexist" in [tags] {  
elasticsearch {  
hosts =\> ["127.0.0.1"]  
index =\> "siae-%{+YYYY.MM.dd}"  
}

# stdout { codec =\> rubydebug }

}  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2018, 2:54pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/6 "2018-04-16T14:54:27Z")

</div>

I suspect the grok is not matching, so you are dropping the message. Can you comment out the drop {} and show us rubydebug output for one of the messages?

---

<div class="post-metadata">

**Author:** ![amritesh\_mishra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amritesh_mishra/32/50251_2.png) [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\
**Post date:** [April 16, 2018, 3:47pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/7 "2018-04-16T15:47:45Z")

</div>

how to get that can you help me on that

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2018, 3:58pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/8 "2018-04-16T15:58:35Z")

</div>

> [@amritesh\_mishra](#):
>
> if "\_grokparsefailure" in [tags] {  
> drop {}  
> }

Put a # at the start of the line that says drop{} and then re-run logstash.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2018, 3:58pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207/9 "2018-05-14T15:58:48Z")

</div>

Closed on May 14, 2018, 3:58pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.