Is it possible to create another index equal to the template?
That's not the default template. It's a default template and it's only matching packetbeat-* indices. If the content you posted there is from GET /_template then it means that you actually don't have a template for logstash-*. In which case just add the template I provided above, but change {"template" : "psm-*"} to {"template" : "logstash-*"}.
But I don't want to delete this template here, packetbeat-*.. I want both to be active..
a PUT /_template will not overwrite the existing resource as long as the resource name does not already exist. In other words, make sure the name of the new logstash template is different than the name of the packetbeat template (which appears to be "packetbeat" here, so you're safe).
I have created a template index but my raw fields are always empty.. I have created the same template posted here.. Why is this?
Per my response in another thread, please start your own thread in Logstash, if that's what you're using to create your fields.
Hi,
I'm new to elasticsearch/logstash/kibana and have all the same novice issue - .raw fields are missing. I've tried all suggestions from this discussion, it didn't help though. I'm using clean setup of logstash v.2.1.1, elasticsearch v2.1.1 and kibana 4.3.1.
Thank you for your help.
I am having the same issue with missing raw fields. In my case the output was indexing to filebeat-date.
I changed this to logstash-date and I can see that the template is similar to the above that should create the raw non-evaluated fields. the default template has also become similar. I do have two differences: there is a fielddata: section under strings mapping that specifies the attribute format:disabled, and I am feeding logstash json formatted data.
Any help you can provide will be much appreciated.
Solved: I apparently had solved the problem by changing the index name to logstash-date.
Even though I had removed the kibana index to force a re-creation, it was necessary to go to kibana settings and update the fields lists - a couple times. i.e. I would look at the fields and could see the raw. but it took a couple clicks on the refresh button before they appeared in the visualize terms aggregation fields list.
all good now.
Same problem. how to set not_analyse for one string feald ??
fluentd - elastic-kibana (syslog input logstash format)
But not clear how it solved in here for such kind index logstash-2016.05.31.
Template mapping is not working....
all day working for it.
Need help .. please.
Hi.
I see similar, that my .raw fields are empty. Is there any solution?
In my case previously i had data in my cluster without raw fields. Later I updated template to get .raw data. Now those fields I can see but those are empty.
Appreciate help.
Thanks
Naming your index in your config to logstash-* will not cause the raw versions to be included if manage_template is set to false in the same logstash config file for the elasticsearch output.
Somewhere in the process of learning I picked the problematic setting up on StackOverflow. I'm sure that guy was doing cool stuff with a custom template, but that's not me.
Here's the corrected output portion of a config file.
output {
elasticsearch {
manage_template => true
index => "logstash-whatever-%{+YYYY.MM.dd}"
}
}
1 Like
I was running into the issue of the .raw fields not appearing due since I was using my own index name. Since there is a pre-built template which covers the "logstash-*" indices, I used that to build a similar template for my new indices. Here are the steps I took to resolve the issue:
Example Logstash Output:
> output {
> elasticsearch {
> hosts => "elasticsearch:9200"
> index => "MyCustomIndex-%{+YYYY.MM.dd}"
> }
> }
What I want to do now is create a template which works similar to the logstash template, so I get a copy of the logstash template from elasticsearch.
curl -XGET localhost:9200/_template/logstash/
{"logstash":{"order":0,"template":"logstash-","settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[{"message_field":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string"},"match_mapping_type":"string","match":"message"}},{"string_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"ignore_above":256,"index":"not_analyzed","type":"string"}}},"match_mapping_type":"string","match":""}}],"_all":{"omit_norms":true,"enabled":true},"properties":{"@timestamp":{"type":"date"},"geoip":{"dynamic":true,"properties":{"ip":{"type":"ip"},"latitude":{"type":"float"},"location":{"type":"geo_point"},"longitude":{"type":"float"}}},"@version":{"index":"not_analyzed","type":"string"}}}},"aliases":{}}}
Then I take that template json and edit it to match the behavior I am looking for; The logstash template covers the the behavior I am looking for, so I simply remove the first scope of the json containing the template name, and update the "template" value to be "MyCustomIndex-*". (don't forget to remove the last '}' )
{"order":0,"template":"MyCustomIndex-*","settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[{"message_field":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string"},"match_mapping_type":"string","match":"message"}},{"string_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"ignore_above":256,"index":"not_analyzed","type":"string"}}},"match_mapping_type":"string","match":"*"}}],"_all":{"omit_norms":true,"enabled":true},"properties":{"@timestamp":{"type":"date"},"geoip":{"dynamic":true,"properties":{"ip":{"type":"ip"},"latitude":{"type":"float"},"location":{"type":"geo_point"},"longitude":{"type":"float"}}},"@version":{"index":"not_analyzed","type":"string"}}}},"aliases":{}}
Now I simply put my updated template into elasticsearch (note that the end of the url is the name I want to give this template). At this point any index with the prefix MyCustomIndex-* will follow the behaviors I laid out in the template.
curl -XPUT 'elasticsearch:9200/_template/MyCustomTemplate?pretty' -d'{"order":0,"template":"MyCustomIndex-","settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[{"message_field":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string"},"match_mapping_type":"string","match":"message"}},{"string_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"ignore_above":256,"index":"not_analyzed","type":"string"}}},"match_mapping_type":"string","match":""}}],"_all":{"omit_norms":true,"enabled":true},"properties":{"@timestamp":{"type":"date"},"geoip":{"dynamic":true,"properties":{"ip":{"type":"ip"},"latitude":{"type":"float"},"location":{"type":"geo_point"},"longitude":{"type":"float"}}},"@version":{"index":"not_analyzed","type":"string"}}}},"aliases":{}}'
Here is some current documentation on templates.
I hope this helps,
-Rob
1 Like
curl -XPUT 'elasticsearch:9200/_template/MyCustomTemplate?pretty' -d'{"order":0,"template":"MyCustomIndex-","settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[{"message_field":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string"},"match_mapping_type":"string","match":"message"}},{"string_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"ignore_above":256,"index":"not_analyzed","type":"string"}}},"match_mapping_type":"string","match":""}}],"all":{"omitnorms":true,"enabled":true},"properties":{"@timestamp":{"type":"date"},"geoip":{"dynamic":true,"properties":{"ip":{"type":"ip"},"latitude":{"type":"float"},"location":{"type":"geo_point"},"longitude":{"type":"float"}}},"@version":{"index":"not_analyzed","type":"string"}}}},"aliases":{}}'
dissertation writing service
If you're sure you have the proper mappings and have .raw fields indexed, one last thing you have to do in Kibana 4 is refresh the fields list under the Settings tab.
Credit to @jeremyjohnreeder for the tip here:
1 Like
I create a new template filebeat-* with the above informations and even if I refresh the indices or delete it and recreate it i don't see any raw fields ! I work with kibana 4.6.
However if i juste create another index in Kibana "filebeat-test-*" which match my logstash index I see raw fields ...
So I'm pretty sure that refres indices isn't enoughI think that the index must be delete in elasticsearch ...
Anyway thanks for the post I can now use kibana vizualisation with filebeat index 
Thank you for your information, i am dissertation writing service. I don't have in programming language but now I have interest in it and I will try to study.