Thanks @leandrojmp for your help. I am now able to create Index accordingly by reading the timestamp value. Could you please help me to understand date_rounding parameter in date_index_name processor.
Thanks,
Debasis
Hi @Debasis_Mallick,
Do you have a particular question on concern on the date_rounding parameter? The possible values and a description of the options you can round the date in the index name with is present in the documentation.
@leandrojmp Now we are able to create automatic Index based the timestamp value mentioned in the record. But I want to incorporate certain additional settings for these Index (example: If want to change datatype of fields, replicas etc ..) . How we achieve the same with this dynamic Index.
Thanks,
Debasis
Hi @leandrojmp Any help on the above request.
Thanks,
Debasis
If you want to control index mappings you need an index template. If you need to change mappings you will need to reindex into a new index with the desired mappings.
But in below pipeline the index is getting created dynamically so where to mention the index template so that it create with proper settings.
PUT _ingest/pipeline/parse_elastic_data_v2
{
"processors": [
{
"csv": {
"description": "Parse elastic Data From CSV Files",
"field": "message",
"target_fields": ["sequence",
"component","tenant",
"service_id","session_id",
"timestamp","edr_version"],
"separator": ",",
"ignore_missing":true,
"trim":true
}
},
{ "date": {
"field": "timestamp",
"target_field": "timestamp_em",
"formats": ["UNIX_MS"]
}
},
"date_index_name": {
"field": "timestamp",
"index_name_prefix": "sfw-",
"date_rounding": "d",
"date_formats": ["UNIX_MS"]
}
}
]
}
Thanks,
Debasis
There is the index_patterns option in the index template that accepts wildcards so you can specify expressions that match your template:
@carly.richmond As I mentioned above , the pipeline is creating a dynamic Index starting with sfw-* as per pipeline index_name_prefix. How we can add Index template in the pipeline. Please advise .
Thanks,
Debasis
Perhaps I'm overly-simplifying things, but would you not specify prefix sfw-* as your index pattern in your index template? If not can you explain why you can't do that?
@carly.richmond Below is my index template and pipeline already mentioned above. The template name I used in filebeat.yml files but still Index are not getting created with the settings what I provided. Could you please help if I am missing anything.
setup.template.name: "sfwidx_template"
setup.template.enabled: true
{
"template": {
"settings": {
"index": {
"number_of_replicas": "1",
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
}
}
},
"mappings": {
"properties": {
"action_id": {
"type": "integer"
},
"component": {
"type": "integer",
"ignore_malformed": false,
"coerce": true
},
"edr_version": {
"type": "keyword"
},
"sequence": {
"type": "keyword"
},
"service_id": {
"type": "integer"
},
"session_id": {
"type": "keyword"
},
"tenant": {
"type": "keyword"
},
"timestamp": {
"type": "date"
}
}
},
"aliases": {
"sfwindex": {}
}
}
}
Thanks,
Debasis
@carly.richmond Any help on this.
Thanks,
Debasis
Hi @Debasis_Mallick,
I'm not sure I follow what you mean. Index templates tie to the index pattern (so a pattern matching the index name) rather than a pipeline. I believe you said you were new to Elastic so I would recommend having a look at the documentation on index templates if you haven't already.
Can you confirm that you've specified the ingest pipeline in your filebeat configuration as covered here? If so, once the index template has also been created you should be reindexing/ reingesting the data using the pipeline to make sure the index template is being applied.
If you think the index is not being applied but it should, you could see if the simulate_index API can help you figure out if the template is being applied or not.
Let us know if that helps!
Hi @carly.richmond
Below two are my requirement.
-
We had implement Filebeat to read the CSV file and generate an index based on the timestamp field within each CSV record dynamically. For instance, if the timestamp (in epoch format) is 1688927400000, the index created will be named
sfw-09-07-2023. This functionality is successfully implemented using the pipelinedate_index_name. -
As the index is generated dynamically based on the timestamp column value, we encounter a limitation in customizing the data type for fields within the index
sfw-09-07-2023. Currently, all fields default to the keyword data type. We are seeking guidance on how to address this issue.
As per your suggestion to use Index templates, we have not been able to resolve this matter.
Thanks,
Debasis
I think that you need to load the template manually, not using the setup.template.* options from filebeat.
Those options expect your index name to follow a pattern like this:
filebeat-FILEBEAT_VERSION
You can have a custom index name, but it would need to follow the same pattern, something like custom_name-8.14.3 for example.
So in your case you will need to create your template and load it manually as mentioned in the documentation shared.
@leandrojmp When I query template , we got the details. Can you help me with the command , how to load the template manually. Sorry to bother you with a simple question, as I am new to Elasticsearch.
GET _index_template/sfwidx_template
{
"index_templates": [
{
"name": "sfwidx_template",
"index_template": {
"index_patterns": [
"sfw-"
],
"template": {
"settings": {
"index": {
"number_of_replicas": "1"
}
},
"mappings": {
"properties": {
"sequence": {
"type": "keyword"
},
"component": {
"coerce": true,
"index": true,
"ignore_malformed": false,
"store": false,
"type": "integer",
"doc_values": true
},
"edr_version": {
"eager_global_ordinals": false,
"norms": false,
"index": true,
"store": false,
"type": "keyword",
"index_options": "docs",
"split_queries_on_whitespace": false,
"doc_values": true
},
"action_id": {
"type": "integer"
},
"service_id": {
"type": "integer"
},
"session_id": {
"type": "keyword"
},
"tenant": {
"type": "keyword"
},
"timestamp": {
"type": "date"
}
}
},
"aliases": {
"sfwindex": {}
}
},
"composed_of": [],
"_meta": {
"description": "sfw Index template"
}
}
}
]
}
@leandrojmp I had tried below command to load Index template manually but getting error. Could you please just verify if the syntax is correct or not.
[root@cb-4 ~]# filebeat setup --index-management -E setup.template.name="sfwidx_template" -E setup.template.pattern="sfw-*" -E 'output.elasticsearch.hosts=["10.10.18.59:9200"]' -E setup.ilm.overwrite=true
Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"},"status":400}. Response body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"},"status":400}
Thanks,
Debasis
Hello,
As mentioned you need to do this manually, not using filebeat.
You need to create and load your template as explained in this documentation.
And also disable template loading in filebeat using:
setup.template.enabled: false
Could you please help me how we can load template manually. I had created the above template using kibana.
Thanks,
Debasis
It is explained in the previous linked documentation.
What does the template you created looks like? Did you use the API through dev tools or using the Kibana UI interface?
To load it manually you juste use:
PUT _index_template/template_name
{
"payload of your template"
}
After that the template is already available and will match your indices?
I had created using Kibana UI interface.
Thanks,
Debasis