Logstash output custom template

venku · October 11, 2020, 4:15am

Hi, I am new to ELK stack. This is my set so far for sending application logs : filebeat -> redis(dockers) -> logstash(dockers) -> ES.
My Question is what is the the correct way to setup custom index in logstash output?
According to my understanding I need to create a index template directly in ES and use the same name in the output of the logstash. Is this not the right approach? Because I don't see the fields are index in the way I defined in the template. Below are steps I followed
Step:

Create a index template in ES using this - _index_template/index2_template and passed JSON with fields types.
Used the same name in the output of the logstash.

JSON for creating index template:

        {
    "index_patterns": ["index2-*"],
      "template": {
      "settings": {
          "number_of_shards": 1
        },
        "mappings": {
          "properties": {
            "timestamp": {
              "type": "date"
            },
    		"level": {
              "type": "keyword"
            },
            "mdc.audit_id": {
              "type": "text",
              "index": false
            },
            "mdc.status": {
              "type": "keyword"
            },
            "mdc.instance_id": {
              "type": "text",
              "index": false
            },
            "mdc.url": {
              "type": "text"
            },
            "mdc.executionTime": {
              "type": "integer"
            },
    		"thread": {
              "type": "keyword",
              "index": false
            },
    		"logger": {
              "type": "text",
              "index": false
            },
    		"message": {
              "type": "keyword"
            },
    		"exception": {
              "type": "text"
            }
          }
        }
      },
      "priority": 51,
      "version": 1,
      "_meta": {
        "description": "my custom"
      }
    }

In the logstash I have the following:

    input {
      redis {
        host => "XXX.1X.X.X"
        key => "app_logs"
        data_type => "list"
      }
    }

    output {
      elasticsearch {
        hosts => ["eslocalhost:9200"]
        index => "index2_template"
        #manage_template => true
        template_name => "index2_template"
    	template => "index2_template"
      }
    }

leandrojmp · October 11, 2020, 5:44pm

Your index pattern in your template does not match your index name.

In your template you have:

"index_patterns": ["index2-*"]

But in your logstash output to elasticsearch you have:

index => "index2_template"

The index index2_template does not match the pattern index2-*, your index name should be something like index2-template to match your template. Or you need to change the patter in your template, it does matter what you will change, but the index name needs to match the index pattern.

venku · October 12, 2020, 5:11pm

Sorry for the delayed reply. I applied the changes you suggested but I am getting an error in logstash. Can you please suggest what else is going wrong here?

Below is the error I see in logstash logs:
[2020-10-12T17:07:45,964][ERROR][logstash.outputs.elasticsearch][main][fc94776fa3282fe5e77f25f9ff0aa9c3626017d4d0bd9ab347a9e4e7847d1e71] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"index3-*", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5c8380c>], :response=>{"index"=>{"_index"=>"index3-*", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [index3-*], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]", "index_uuid"=>"_na_", "index"=>"index3-*"}}}}

my template:

{
"index_patterns": ["index3-*"],
  "template": {
  "settings": {
      "number_of_shards": 1,
      "index.lifecycle.name": "ecom_ilm_policy",
      "index.lifecycle.rollover_alias":"ecom"
    },
    "mappings": {
      "properties": {
        "timestamp": {
          "type": "date"
        },
		"level": {
          "type": "keyword"
        },
        "mdc.audit_id": {
          "type": "text",
          "index": false
        },
        "mdc.status": {
          "type": "keyword"
        },
        "mdc.instance_id": {
          "type": "text",
          "index": false
        },
        "mdc.url": {
          "type": "text"
        },
        "mdc.executionTime": {
          "type": "integer"
        },
		"thread": {
          "type": "keyword",
          "index": false
        },
		"logger": {
          "type": "text",
          "index": false
        },
		"message": {
          "type": "keyword"
        },
		"exception": {
          "type": "text"
        }
      }
    }
  },
  "priority": 51,
  "version": 1,
  "_meta": {
    "description": "my custom"
  }
}

Logstash Config

input {
  redis {
    host => "172.17.0.2"
    key => "ecom"
    data_type => "list"
  }
}

output {
  elasticsearch {
    hosts => ["eslocalhost:9200"]
    index => "index3-*"
    # manage_template => true
    template_name => "index3_template"
    ilm_enabled => true

  }
}

Badger · October 12, 2020, 5:20pm

You cannot call an index "index3-*". You can use that in the index_patterns field of the template, but you cannot use it in the index option of an elasticsearch output. What do you want the index to be called?

venku · October 12, 2020, 5:24pm

I want my index be called as "ecom-app-logs". And will use ecom* to filter all indices in Kibana is the plan.

Badger · October 12, 2020, 5:35pm

OK, so in logstash use

output {
    elasticsearch {
        hosts => ["eslocalhost:9200"]
        index => "ecom-app-logs"

and in your template use

"index_patterns": ["ecom-*"],

venku · October 12, 2020, 5:40pm

Thank you. I think it is making sense to me now.
Quick question before I make the change, do I have to change anything fortemplate_name. What is this used for please?

Thanks!

venku · October 12, 2020, 5:59pm

I made the change and tried to create an index in Kibana, I am not sure if the template is picked up. Because, I marked the level as a keyword but I see it as string in Kibana

"level": {
          "type": "keyword"
        },

Badger · October 12, 2020, 6:48pm

templates are stored in elasticsearch. They need a name so that you can refer to it if you want to overwrite it. You do not need to change the template name but some people prefer to have a template name that is similar to the index names that it applies to.

If you want to know what mapping (not template) was actually applied to an index then use the mapping API.

Prior to logstash 5.0 there was a string type in elasticsearch. In 5.0 strings were replaced with the two types text and keyword. However, I do not think Kibana was update to call them that.

I believe that your template was applied, because Kibana shows the field as aggregatable, which would not be true if it were text.

venku · October 12, 2020, 7:20pm

@Badger, You are correct. Looks like the template is applied. Thank you again for all the help and insights.

system · November 9, 2020, 7:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problem with my personal index creation for platform logs Logstash	1	157	September 9, 2022
Custom index template is not working Elasticsearch	2	948	March 6, 2019
Set a template_name in Logstash output filter to send data elasticsearch Logstash	3	2562	December 31, 2019
Custom Template for Logstash 7.10 output Logstash	1	1005	February 18, 2021
Pointing output.conf to a custom added template Logstash	4	853	July 6, 2017

Logstash output custom template

Related topics