Index not returning results - ELK - Can Index creation be automated?

Elastic Stack Logstash
1

Hey users
Been trying to figure out below issue but not successful. I created an index to match a data pattern in elasticsearch - but not fetching results when i discover

Results are displaying for default logstash-* pattern. What am i missing here?
Earlier logstash version created index automatically. Can index creation be automated rather than end users use Postman plugin to create indexes ?

Data -

{
"message" => "{"message":"2016-07-04T02:34:37.642-0700: 505452.911: [GC (Allocation Failure) [PSYoungGen: 340349K->4283K(342016K)] 754917K->418851K(1339904K), 0.0130968 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] ","@version":"1","@timestamp":"2016-07-04T09:34:38.098Z","host":"SERVER","path":"C:/instances/default/logs/gc.log","type":"Server.gc","environment":"DEVELOPMENT"}",
"@version" => "1",
"@timestamp" => "2016-07-04T09:34:38.145Z",
"application" => "Server",
"dimension" => "JVM Garbage Collection",
"timestamp" => "2016-07-04T02:34:37.642-0700",
"elapse" => 505452.911,
"gctype" => "GC",
"gctext" => "(Allocation Failure)",
"newbefore" => 340349,
"newafter" => 4283,
"newmax" => 342016,
"heapbefore" => 754917,
"heapafter" => 418851,
"heapmax" => 1339904,
"gctime" => 0.0130968
}

Index created

{
"mappings": {
"default": {
"properties": {
"message": {
"type": "string"
},
"@version": {
"type": "string", "index" : "not_analyzed"
},
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"application": {
"type": "string", "index" : "not_analyzed"
},
"dimension": {
"type": "string", "index" : "not_analyzed"
},
"timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"elapse": {
"type": "double", "index" : "not_analyzed"
},
"gctype": {
"type": "string", "index" : "not_analyzed"
},
"gctext": {
"type": "string", "index" : "not_analyzed"
},
"newbefore": {
"type": "long"
},
"newafter": {
"type": "long"
},
"newmax": {
"type": "long"
},
"heapbefore": {
"type": "long"
},
"heapafter": {
"type": "long"
},
"heapmax": {
"type": "long"
},
"gctime": {
"type": "double", "index" : "not_analyzed"
}
}
}
}
}

Create the index in elastic search through postman plugin
When i created the index in Kibana and did Discover - It states "No results found"

2

Earlier logstash version created index automatically.

That hasn't changed in recent versions. Well, it's actually ES that creates the indexes as needed.

3

Hi Magnus

You are right. I created the index in ES and then i have to Configure that index in Kibana.

When i discover - It states No results returned

Regards
Vinay

4

Never mind Kibana for now. Does the index exist? Does it contain any documents?

5

The index exists now. I created the index in elastic search and visible in http://localhost:9200/_cat/indices?v

But it does not contain any documents. You can refer my post for index created

Regards
Vinay

6

Is Logstash having problems posting documents to the index? The Logstash log should tell you. Is Logstash even processing any events, i.e. are your inputs working correctly?

7

Yes magnus
My inputs are working correctly i dont see any errors in logstash.
The posted documents reflects correctly in default index logstash-*
It even reflects correctly in daily log index logstash-2016.09.14. When i search in daily log index it does shows the documents which i posted.

But the newly created index does not fetch any results.

Regards
Vinay

8

What does your configuration look like?

9

OS - Windows server 2012
Elastic search 2.1.1
Logstash 2.3.4
Kibana 4.3.1
ActiveMQ for Message provider

10

Technically i dont see any issue (data types, mappings etc) with the index which i have created. Please check and confirm

11

Your Logstash configuration. What's in your Logstash configuration file(s)?

12

Hi magnus
Got delayed in reply.
Please find below the logstash configuration files
Shipper and Indexer in github repo

Regards
Vinay

13

Hi Magnus

I have uploaded the two index contents in document. Please have a look and help

indexNotReadingDocuments-Brief.docx

Regards
Vinay

14

Team

Can you check and reply on this pending item

Regards
Vinay

15

I saw your response earlier but for some reason didn't respond. Maybe I didn't know what was wrong.