Logstash output custom template

Elastic Stack Logstash
1

Hi, I am new to ELK stack. This is my set so far for sending application logs : filebeat -> redis(dockers) -> logstash(dockers) -> ES.
My Question is what is the the correct way to setup custom index in logstash output?
According to my understanding I need to create a index template directly in ES and use the same name in the output of the logstash. Is this not the right approach? Because I don't see the fields are index in the way I defined in the template. Below are steps I followed
Step:

  1. Create a index template in ES using this - _index_template/index2_template and passed JSON with fields types.
  2. Used the same name in the output of the logstash.

JSON for creating index template:

        {
    "index_patterns": ["index2-*"],
      "template": {
      "settings": {
          "number_of_shards": 1
        },
        "mappings": {
          "properties": {
            "timestamp": {
              "type": "date"
            },
    		"level": {
              "type": "keyword"
            },
            "mdc.audit_id": {
              "type": "text",
              "index": false
            },
            "mdc.status": {
              "type": "keyword"
            },
            "mdc.instance_id": {
              "type": "text",
              "index": false
            },
            "mdc.url": {
              "type": "text"
            },
            "mdc.executionTime": {
              "type": "integer"
            },
    		"thread": {
              "type": "keyword",
              "index": false
            },
    		"logger": {
              "type": "text",
              "index": false
            },
    		"message": {
              "type": "keyword"
            },
    		"exception": {
              "type": "text"
            }
          }
        }
      },
      "priority": 51,
      "version": 1,
      "_meta": {
        "description": "my custom"
      }
    }

In the logstash I have the following:

    input {
      redis {
        host => "XXX.1X.X.X"
        key => "app_logs"
        data_type => "list"
      }
    }

    output {
      elasticsearch {
        hosts => ["eslocalhost:9200"]
        index => "index2_template"
        #manage_template => true
        template_name => "index2_template"
    	template => "index2_template"
      }
    }
2

Your index pattern in your template does not match your index name.

In your template you have:

"index_patterns": ["index2-*"]

But in your logstash output to elasticsearch you have:

index => "index2_template"

The index index2_template does not match the pattern index2-*, your index name should be something like index2-template to match your template. Or you need to change the patter in your template, it does matter what you will change, but the index name needs to match the index pattern.

3

Sorry for the delayed reply. I applied the changes you suggested but I am getting an error in logstash. Can you please suggest what else is going wrong here?

Below is the error I see in logstash logs:
[2020-10-12T17:07:45,964][ERROR][logstash.outputs.elasticsearch][main][fc94776fa3282fe5e77f25f9ff0aa9c3626017d4d0bd9ab347a9e4e7847d1e71] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"index3-*", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5c8380c>], :response=>{"index"=>{"_index"=>"index3-*", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [index3-*], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]", "index_uuid"=>"_na_", "index"=>"index3-*"}}}}

my template:

{
"index_patterns": ["index3-*"],
  "template": {
  "settings": {
      "number_of_shards": 1,
      "index.lifecycle.name": "ecom_ilm_policy",
      "index.lifecycle.rollover_alias":"ecom"
    },
    "mappings": {
      "properties": {
        "timestamp": {
          "type": "date"
        },
		"level": {
          "type": "keyword"
        },
        "mdc.audit_id": {
          "type": "text",
          "index": false
        },
        "mdc.status": {
          "type": "keyword"
        },
        "mdc.instance_id": {
          "type": "text",
          "index": false
        },
        "mdc.url": {
          "type": "text"
        },
        "mdc.executionTime": {
          "type": "integer"
        },
		"thread": {
          "type": "keyword",
          "index": false
        },
		"logger": {
          "type": "text",
          "index": false
        },
		"message": {
          "type": "keyword"
        },
		"exception": {
          "type": "text"
        }
      }
    }
  },
  "priority": 51,
  "version": 1,
  "_meta": {
    "description": "my custom"
  }
}

Logstash Config

input {
  redis {
    host => "172.17.0.2"
    key => "ecom"
    data_type => "list"
  }
}

output {
  elasticsearch {
    hosts => ["eslocalhost:9200"]
    index => "index3-*"
    # manage_template => true
    template_name => "index3_template"
    ilm_enabled => true

  }
}
4

You cannot call an index "index3-*". You can use that in the index_patterns field of the template, but you cannot use it in the index option of an elasticsearch output. What do you want the index to be called?

5

I want my index be called as "ecom-app-logs". And will use ecom* to filter all indices in Kibana is the plan.

6

OK, so in logstash use

output {
    elasticsearch {
        hosts => ["eslocalhost:9200"]
        index => "ecom-app-logs"

and in your template use

"index_patterns": ["ecom-*"],
7

Thank you. I think it is making sense to me now.
Quick question before I make the change, do I have to change anything fortemplate_name. What is this used for please?

Thanks!

8

I made the change and tried to create an index in Kibana, I am not sure if the template is picked up. Because, I marked the level as a keyword but I see it as string in Kibana

"level": {
          "type": "keyword"
        },

kibana_level
kibana_level1218×485 29 KB

9

templates are stored in elasticsearch. They need a name so that you can refer to it if you want to overwrite it. You do not need to change the template name but some people prefer to have a template name that is similar to the index names that it applies to.

If you want to know what mapping (not template) was actually applied to an index then use the mapping API.

Prior to logstash 5.0 there was a string type in elasticsearch. In 5.0 strings were replaced with the two types text and keyword. However, I do not think Kibana was update to call them that.

I believe that your template was applied, because Kibana shows the field as aggregatable, which would not be true if it were text.

1 Like
10

@Badger, You are correct. Looks like the template is applied. Thank you again for all the help and insights.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.