Automatic Index Creation

Thanks @leandrojmp for your help. I am now able to create Index accordingly by reading the timestamp value. Could you please help me to understand date_rounding parameter in date_index_name processor.

Thanks,
Debasis

Hi @Debasis_Mallick,

Do you have a particular question on concern on the date_rounding parameter? The possible values and a description of the options you can round the date in the index name with is present in the documentation.

@leandrojmp Now we are able to create automatic Index based the timestamp value mentioned in the record. But I want to incorporate certain additional settings for these Index (example: If want to change datatype of fields, replicas etc ..) . How we achieve the same with this dynamic Index.

Thanks,
Debasis

Hi @leandrojmp Any help on the above request.

Thanks,
Debasis

If you want to control index mappings you need an index template. If you need to change mappings you will need to reindex into a new index with the desired mappings.

But in below pipeline the index is getting created dynamically so where to mention the index template so that it create with proper settings.

PUT _ingest/pipeline/parse_elastic_data_v2
{
  "processors": [
    {
      "csv": {
        "description": "Parse elastic Data From CSV Files",
        "field": "message",
        "target_fields": ["sequence",
        "component","tenant",
        "service_id","session_id",
        "timestamp","edr_version"],
        "separator": ",",
        "ignore_missing":true,
        "trim":true
      }
      },
	  { "date": {
          "field": "timestamp",
          "target_field": "timestamp_em",
          "formats": ["UNIX_MS"]
        }
	     },
		 
      "date_index_name": {
        "field": "timestamp",
        "index_name_prefix": "sfw-",
        "date_rounding": "d",
        "date_formats": ["UNIX_MS"]
      }
    }
 ]
}

Thanks,
Debasis

There is the index_patterns option in the index template that accepts wildcards so you can specify expressions that match your template:

@carly.richmond As I mentioned above , the pipeline is creating a dynamic Index starting with sfw-* as per pipeline index_name_prefix. How we can add Index template in the pipeline. Please advise .

Thanks,
Debasis

Perhaps I'm overly-simplifying things, but would you not specify prefix sfw-* as your index pattern in your index template? If not can you explain why you can't do that?

@carly.richmond Below is my index template and pipeline already mentioned above. The template name I used in filebeat.yml files but still Index are not getting created with the settings what I provided. Could you please help if I am missing anything.

setup.template.name: "sfwidx_template"
setup.template.enabled: true
{
  "template": {
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "action_id": {
          "type": "integer"
        },
        "component": {
          "type": "integer",
          "ignore_malformed": false,
          "coerce": true
        },
        "edr_version": {
          "type": "keyword"
        },
        "sequence": {
          "type": "keyword"
        },
        "service_id": {
          "type": "integer"
        },
        "session_id": {
          "type": "keyword"
        },
        "tenant": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date"
        }
      }
    },
    "aliases": {
      "sfwindex": {}
    }
  }
}

Thanks,
Debasis

@carly.richmond Any help on this.

Thanks,
Debasis

Hi @Debasis_Mallick,

I'm not sure I follow what you mean. Index templates tie to the index pattern (so a pattern matching the index name) rather than a pipeline. I believe you said you were new to Elastic so I would recommend having a look at the documentation on index templates if you haven't already.

Can you confirm that you've specified the ingest pipeline in your filebeat configuration as covered here? If so, once the index template has also been created you should be reindexing/ reingesting the data using the pipeline to make sure the index template is being applied.

If you think the index is not being applied but it should, you could see if the simulate_index API can help you figure out if the template is being applied or not.

Let us know if that helps!

Hi @carly.richmond
Below two are my requirement.

  1. We had implement Filebeat to read the CSV file and generate an index based on the timestamp field within each CSV record dynamically. For instance, if the timestamp (in epoch format) is 1688927400000, the index created will be named sfw-09-07-2023. This functionality is successfully implemented using the pipeline date_index_name.

  2. As the index is generated dynamically based on the timestamp column value, we encounter a limitation in customizing the data type for fields within the index sfw-09-07-2023. Currently, all fields default to the keyword data type. We are seeking guidance on how to address this issue.

As per your suggestion to use Index templates, we have not been able to resolve this matter.

Thanks,
Debasis

I think that you need to load the template manually, not using the setup.template.* options from filebeat.

Those options expect your index name to follow a pattern like this:

filebeat-FILEBEAT_VERSION

You can have a custom index name, but it would need to follow the same pattern, something like custom_name-8.14.3 for example.

So in your case you will need to create your template and load it manually as mentioned in the documentation shared.

@leandrojmp When I query template , we got the details. Can you help me with the command , how to load the template manually. Sorry to bother you with a simple question, as I am new to Elasticsearch.

GET _index_template/sfwidx_template

{
  "index_templates": [
    {
      "name": "sfwidx_template",
      "index_template": {
        "index_patterns": [
          "sfw-"
        ],
        "template": {
          "settings": {
            "index": {
              "number_of_replicas": "1"
            }
          },
          "mappings": {
            "properties": {
              "sequence": {
                "type": "keyword"
              },
              "component": {
                "coerce": true,
                "index": true,
                "ignore_malformed": false,
                "store": false,
                "type": "integer",
                "doc_values": true
              },
              "edr_version": {
                "eager_global_ordinals": false,
                "norms": false,
                "index": true,
                "store": false,
                "type": "keyword",
                "index_options": "docs",
                "split_queries_on_whitespace": false,
                "doc_values": true
              },
              "action_id": {
                "type": "integer"
              },
              "service_id": {
                "type": "integer"
              },
              "session_id": {
                "type": "keyword"
              },
              "tenant": {
                "type": "keyword"
              },
              "timestamp": {
                "type": "date"
              }
            }
          },
          "aliases": {
            "sfwindex": {}
          }
        },
        "composed_of": [],
        "_meta": {
          "description": "sfw Index template"
        }
      }
    }
  ]
}