ILM settings not being applied

Elastic Stack Elasticsearch
1

I have a logstash index storing data that will be used in reporting, but only ever for the last 5 minutes.
Hence, I want to have a very aggressive deletion policy.
Roll the index every 15 minutes
Delete the rolled (old) index.

However, my ILM, index template and index creation must have a flaw in it, because every time I view the ILM status of my index, my settings aren't there.

Can someone please point out where I'm going wrong?

ILM POLICY

PUT /_ilm/policy/logstash-policy
{
   "policy": {
     "phases": {
       "hot": {
         "actions": {
           "rollover": {
             "max_age": "15m",
             "max_size": "10GB"
           }
         }
       },
       "delete": {
          "min_age": "15m",
          "actions": {
          "delete": {}
        }
      }
    }
  }
}

INDEX TEMPLATE

PUT /_template/logstash_template
{
  "index_patterns": ["logstash-*"],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index.lifecycle.name": "logstash-policy",
    "index.lifecycle.rollover_alias": "logstash-alias"
  }
}

INDEX CREATION

PUT /logstash-000001 
{
    "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "aliases": {
      "logstash":{
        "is_write_index": true
        }
   },
  "mappings": {
                      <various field mappings>
  }
}

And when I query the ILM settings, I don't see the things I expect.

GET logstash*/_ilm/explain?pretty
{
  "indices" : {
    "logstash-000001" : {
      "index" : "logstash-000001",
      "managed" : true,
      "policy" : "logstash-policy",
      "lifecycle_date_millis" : 1596507802885,
      "age" : "1.16h",
      "phase" : "hot",
      "phase_time_millis" : 1596507802959,
      "action" : "rollover",
      "action_time_millis" : 1596508792079,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1596508792079,
      "phase_execution" : {
        "policy" : "logstash-policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1595979741503
      }
    }
  }
}

The max_size and max_age values look like they came from a default somewhere.

Can someone spot where I'm going wrong?

2

I think I found it, the problem seems to have been in the index creation code. I should have put aliases.logstash-alias.is_write_index instead of aliases.logstash.is_write_index

INDEX CREATION

PUT /logstash-000001 
{
    "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "aliases": {
      "logstash-alias":{
        "is_write_index": true
        }
   },
  "mappings": {
                      <various field mappings>
  }
}

However, that's brought a new issue to light. In the field mappings, I define a field as type=ip so I can do CIDR searches. Something, I suspect logstash, is changing that back to a type=text.
Here's my logstash config

input {
  tcp {
    port => 10514
    type => bc_syslog
  }
}

filter {
    # drop comment lines
    if ([message] =~ /^#/) {
      drop{}
    }
    csv {
        columns => ["date", "time", "time-taken", "c-ip", "cs-username", "cs-auth-group", "s-supplier-name", "s-supplier-ip", "s-supplier-country", "s-supplier-failures", "x-exception-id", "sc-filter-result", "cs-categories", "cs-Referer", "sc-status", "s-action", "cs-method", "rs-Content-Type", "cs-uri-scheme", "cs-host", "cs-uri-port", "cs-uri-path", "cs-uri-query", "cs-uri-extension", "cs-User-Agent", "s-ip", "sc-bytes", "cs-bytes", "x-virus-id", "x-bluecoat-application-name", "x-bluecoat-application-operation", "r-ip", "cs-threat-risk", "x-bluecoat-transaction-uuid", "x-icap-reqmod-header-X-ICAP-Metadata", "x-icap-respmod-header-X-ICAP-Metadata", "x-rs-certificate-validate-status", "x-rs-certificate-observed-errors", "x-cs-ocsp-error", "x-rs-ocsp-error", "x-rs-connection-negotiated-ssl-version", "x-rs-connection-negotiated-cipher", "x-rs-connection-negotiated-cipher-size", "x-rs-certificate-hostname", "x-rs-certificate-hostname-category", "x-cs-connection-negotiated-ssl-version", "x-cs-connection-negotiated-cipher", "x-cs-connection-negotiated-cipher-size", "x-cs-certificate-subject", "s-sitename", "x-rs-certificate-hostname-threat-risk", "rs-bytes", "c-starttime", "filelength", "filesize", "avgbandwidth", "x-rs-streaming-content", "x-streaming-rtmp-app-name", "x-streaming-rtmp-stream-name", "x-streaming-rtmp-swf-url", "x-streaming-rtmp-page-url", "s-dns", "s-session-id", "x-cache-info", "s-port"]
        separator => " "
    }
    # parse timestamp
    if [date] and [time] {
        mutate {
            add_field => { "timestamp" => "%{date} %{time}" }
        }
        date {
            match => ["timestamp", "YYYY-MM-dd HH:mm:ss" ]
            timezone => ['Australia/Queensland']
        }
    }
	

}

output {
  elasticsearch { hosts => ["localhost:9200"]
        ilm_rollover_alias => "logstash-alias"
        ilm_pattern => "000001"
        ilm_policy => "logstash-policy"
  }
}

Am I missing something in logstash to tell it to leave the index fields alone?

3

I'd recommend you raise a new issue for this, since it looks like you have resolved the ilm issue and this is now about field mappings.

That said, have you considered using the Ingest Pipeline functionality in Elasticsearch to do the CSV extraction, rather than Logstash?

https://www.elastic.co/guide/en/elasticsearch/reference/master/csv-processor.html

And https://www.elastic.co/blog/indexing-csv-elasticsearch-ingest-node

This would give you more control at Elasticsearch, and remove some dependency on Logstash.

If you are still having issues, in a new Topic please share:

a) You're mapping template, and

b) an example raw event

And we can delve a bit deeper!

Stuart.

4

Cheers Stuart, but again I think I found what I was doing wrong. I didn't realise logstash was creating the indexes (according to the template), so my field definitions should've gone in the template, not the index.

5

Great! Glad you got it fixed - yes, the index template is where these need to be defined! Pleased its working for you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.