Fields not saved with template mapping

Elastic Stack Elasticsearch
1

Hi,

Before now I was using dynamic templates as it is quite easy to structure the data automatically and in case of needing new fields this is done automatically.

Now that I have every field that I need, I want to define the field types to improve everything (searcheability, storage, performance...)

I have taken the dynamic template as example and redefine the field types to the needs. The template is stored in the right way but now I am not getting the field only basic data.

Logstash is not trowing any error of mapping.

Logstash output

{
"fac" => "f_dns_proxy",
"area" => "a_aclquery",
"cmd" => "dnsp",
"dstport" => 53,
"rule_name" => "my rule",
"dstip" => "1.1.1.1",
"host" => "0.0.0.0",
"@version" => "1",
"reason" => "Traffic allowed by policy.",
"logid" => 0,
"srcport" => 55890,
"hostname" => "example.example.org",
"srczone" => "internal",
"pri" => "p_major",
"syslog5424_pri" => "46",
"@timestamp" => 2019-11-06T11:27:13.698Z,
"event" => "ACL allow",
"dstzone" => "internal",
"pid" => "32445",
"type" => "t_aclallow",
"application" => "DNS\n",
"program" => "auditd",
"srcip" => "1.1.1.1",
"cache_hit" => 1,
"protocol" => "17",
"logsource" => "example"
}

WIth dynamic everything fine

Dynamic
Dynamic.PNG1324×106 24.4 KB

Using template I am not seeing any field any longer

template
template.PNG1497×303 16.6 KB

2

I was not able to to attach the template because the character limitations.

Template

[PUT _template/fwsw_1
{
"index_patterns": ["logstash-firewall-syslog-sidewinder-*"],
"settings": {
"number_of_shards": 5
},
"mappings": {
"_source": {
"enabled": false
},
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"app_categories": {
"type": "keyword"
},
"app_risk": {
"type": "keyword"
},
"application": {
"type": "keyword"
},
"area": {
"type": "keyword"
},
"attackip": {
"type": "ip"
},
"attackzone": {
"type": "keyword"
},
"bytes_written_to_client": {
"type": "long"
},
"bytes_written_to_server": {
"type": "long"
},
"cache_hit": {
"type": "integer"
},
"category": {
"type": "keyword"
},
"cky_i": {
"type": "keyword"
},
"cky_r": {
"type": "keyword"
},
"cmd": {
"type": "keyword"
},
"config_area": {
"type": "keyword"
},
"config_item": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"cpu_data": {
"type": "integer"
},
"dst_geo": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dstip": {
"type": "ip"
},
"dstport": {
"type": "integer"
},
"dstzone": {
"type": "keyword"
},
"event": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"fac": {
"type": "keyword"
},
"geoip": {
"dynamic": "true",
"properties": {
"city_name": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"continent_code": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code2": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_code3": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"country_name": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
},
"postal_code": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_code": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"region_name": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timezone": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"host": {
"type": "keyword"
},
"hostname": {
"type": "keyword"
},
"ibytes": {
"type": "long"
},
"information": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"interface": {
"type": "keyword"
},
"ipkt": {
"type": "integer"
},
"load_data": {
"type": "integer"
},
"local_gw": {
"type": "ip"
},
"local_net": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"logid": {
"type": "integer"
},
"logsource": {
"type": "keyword"
},
"mbuf_data": {
"type": "integer"
},
"msg_id": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"netsessid": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"obytes": {
"type": "long"
},
"opkt": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"pid": {
"type": "keyword"
},
"pri": {
"type": "keyword"
},
"program": {
"type": "keyword"
},
"protocol": {
"type": "keyword"
},
"real_data": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"reason": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"remote_gw": {
"type": "ip"
},
"remote_id": {
"type": "keyword"
},
"remote_logname": {
"type": "keyword"
},
"remote_net": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"request_command": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"rule_name": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"spi": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src_geo": {
"type": "keyword"
},
"srcip": {
"type": "ip"
},
"srcport": {
"type": "integer"
},
"srczone": {
"type": "keyword"
},
"ssl_name": {
"type": "keyword"
},
"syslog5424_pri": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"total_events": {
"type": "integer"
},
"type": {
"type": "keyword"
},
"udb_action": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"udb_class": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"udb_user": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user_name": {
"type": "keyword"
},
"virt_data": {
"type": "integer"
},
"vpn_name": {
"type": "keyword"
}
}
}
}

3

FEEDBACK

I created the template from the GUI following the "Create template" wizard and now it is working. Probably something wasn't defined properly.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.