Dynamic Index in ElasticSearch

Elastic Stack Logstash
1

I have following configuration in logstash whereby I am able to create dynamic "document_type" into ES based on input JSON received:

    elasticsearch {
    			hosts => ["localhost:9200"]
    			index => "queuelogs"
    			document_type => "%{action}"
    		}

Here, "action" is the parameter that I receive in JSON and different document_type gets created as per different action received.

Now I want this to be done same for Index creation, such as following:

elasticsearch {
			hosts => ["localhost:9200"]
			index => "%{logtype}"
			document_type => "%{action}"
		}

Here, "logtype" is the parameter that I receive in JSON.

But somehow in ES, it creates index as "%{logtype}" only, not as per actual logtype value .

2

Are you sure %{logtype} is being created/populated?

3

The input JSON is as following:

{
  "action": "UPLOAD",
  "user": "123",
  "timestamp": "2016 Jun 14 12:00:12",
  "data": {
    "file_id": "2345",
    "file_name": "xyz.pdf"
  },
  "header": {
    "proj_id": "P123",
    "logtype": "httplogs"
  },
  "comments": "Check comments"
}

Here, I tried to generate index in following ways:

  1. index => "%{logtype}"
  2. index => "%{header.logtype}"

But in both the cases, Logstash does not replace the actual value of logtype from JSON.

4

I was able to solve the problem using following approach:

elasticsearch {
			hosts => ["localhost:9200"]
			index => "%{[header][logtype]}"
			document_type => "%{action}"
		}