Logstash 7.10 is ignoring custom index template

[ManuelF (Manuel)]

Great! It's a good start. Thank you.

Now I would like to focus on that line, because I have read about different ways to send a .csv file to ELK (Logstash, Filebeat, ES Node Ingest Pipelines, etc...), but I want to focus in just one way.

Can you please check my config file below, where I'm using just the CSV filter and not the CSV codec, and see if there is something wrong with it or if I am missing anything or using deprecated names/configs?

I just noticed that when I run Logstash, the index template included in the config template_name => "ts_reports" it is added but empty. I delete it every time and every time it's being added again, so that's working, but for some unknown reason for me, it is not including the template body.

Perhaps that's what is failing. If you can, please take a look as well.

Thank you in advance

ts_reports.conf

input {
    file {
        id => "TS_Reports"
        path => "/opt/ts_reports/*.csv"
#        codec => "csv"
        mode => "read"
        start_position => "beginning"
        file_completed_action => "delete"
        type => "TS"
    }
}
filter {
    csv {
        columns => [
                "Time",
                "Device",
                "Source IP",
                "Source Port",
                "Destination IP",
                "Destination Port",
                "Action",
                "Direction",
                "Targets",
                "ID"
        ]
        separator => ","
        }
}
output {
     elasticsearch {
        hosts => ["https://127.0.0.1:9200"]
        index => "ts_reports-%{+YYYY.MM}"
        manage_template => true
        template => "/etc/logstash/ts_reports-template.json"
        template_name => "ts_reports"
        user => "logstash_internal"
        password => "[PASSWORD]"
        ssl => true
        ssl_certificate_verification => true
        cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
    }
}

ts_reports-template.json

{
  "index_patterns" : ["ts_reports-*"],
  "template": {
    "settings": {
      "index.number_of_replicas": 0,
      "index.refresh_interval" : "5s"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
         },
         "Action": {
           "type": "keyword"
         },
         "Destination IP": {
           "type": "ip"
         },
         "Destination Port": {
           "type": "long"
         },
         "Device": {
           "type": "keyword"
         },
         "Direction": {
           "type": "keyword"
         },
         "ID": {
           "type": "keyword"
         },
         "Source IP": {
           "type": "ip"
         },
         "Source Port": {
           "type": "long"
         },
         "Targets": {
           "type": "keyword"
         },
         "Time": {
           "type": "date",
           "format": "yyyy-MM-dd HH:mm:ss"
         }
      }
    }
  }
}

pipelines.yaml

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: ts_reports
  path.config: "/etc/logstash/conf.d/ts_reports/ts_reports.conf"

[ManuelF (Manuel)]

Hi, @aaron-nimocks

Did you have any chance to check my config or templates?

Thank you

[aaron-nimocks]

I have never used those options so it might be best to start a new topic and ask that specific question again.

Typically I just create the template in Elastic and do nothing on the logstash side.

[ManuelF (Manuel)]

Hi. @aaron-nimocks,

I don't have a preference for a specific route, or yes, the one that works . I could try to do it like you and create the template manually in ES and not in Logstash. The problem is that if Logstash does not find a template for the current pipeline, it will try to apply the default template, which is essentially what has been happening so far.

When Logstash loads the configuration file for this pipeline, it even manages to create an index template with the specified name, but this template is empty, so only the line referring to the name works, but not the .json template. This is why I think the problem lies in the .json template structure.

I have added the index template from the Dev console and it works perfectly. I get the "acknowledged": true. Then when reviewing the template, it has all the expected fields and types.

So, can you please help me configure Logstash so that it doesn't use the default template and does use the index template that already exists in ES?

Thank you

[ManuelF (Manuel)]

Hi @aaron-nimocks,

By just disabling the following lines in the pipeline config would be enough so Logstash use the template that I installed manually in ES, and not the default template?:

    manage_template => true
    template => "/etc/logstash/ts_reports-template.json"
    template_name => "ts_reports"

[ManuelF (Manuel)]

Hi @aaron-nimocks,

Can you please help me configure Logstash to use your way?

Thank you

[aaron-nimocks]

You can remove those 3 above.

If you create your template in Elastic then you are good to go.