Some of the Fortigate fields are not searchable?

[Long_Thai]

I am using kv() to identify key/value pairs. The stream of inputs are from /var/log/syslog and filbeat.
I rename some of the fields like type, subtype, user, group, service to fgt_type, fgt_subtype, fgt_user, fgt_group, fgt_service.

<
kv {
source => "syslog_message"
field_split => ","
value_split => "="
exclude_keys => [ "devid", "devname", "logid", "vd", "poluuid", "sessionid", "sentpkt","rcvdpkt", "crscore", "crlevel" ]
}
if [syslog_hostname] == "a.b.c.d" {
mutate {
rename => { "type" => "fgt_type" }
rename => { "user" => "fgt_user" }
rename => { "subtype" => "fgt_subtype" }
add_field => { "type" => "fortigate" }
rename => { "service" => "fgt_service" }
rename => { "group" => "fgt_group" }
rename => { "url" => "fgt_url" }
remove_field =>["date","time","devname","devid","logid","vd","eventtime","sentpkt","rcvdpkt","poluuid","host.id"]
}
geoip {
source => "srcip"
}
}
/>

But in Elasticsearch, we find that some of the Fortigate fields are not searchable such as user, group, etc.

When we look at the mappings between searchable fields and non-searchable fields, they look the same.

fgt_type is searchable:
<
GET /logstash-filebeat-rsyslog_data-2019.10.25/_mapping/field/fgt_type
{
"logstash-filebeat-rsyslog_data-2019.10.25" : {
"mappings" : {
"fgt_type" : {
"full_name" : "fgt_type",
"mapping" : {
"fgt_type" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
/>

fgt_user is not searchable

<
GET /logstash-filebeat-rsyslog_data-2019.10.25/_mapping/field/fgt_user
{
"logstash-filebeat-rsyslog_data-2019.10.25" : {
"mappings" : {
"fgt_user" : {
"full_name" : "fgt_user",
"mapping" : {
"fgt_user" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
/>

What should we do in order for fgt_user or user becomes searchable in Elasticsearch ?

Thanks

[Christian_Dahlqvist]

What is the query you run to determine that the field is not searchable? Can you show a document from that index that you would expect to match? Are you getting any error message?

[Long_Thai]

Hi,

In the KQL box, I tried to search for fgt_user:* and it gives the results in the attached fgt_user_table file.

If I look through all the fields, it indicates that key fg_user as "?" which says "unable to filter for presence of meta fields"

I list here the message where all key value pairs are in son format:

<

{
"_index": "logstash-filebeat-rsyslog_data-2019.10.27" ,
"_type": "_doc" ,
"_id": "074mDW4BuJS9LxnKYIIb" ,
"_version": 1 ,
"_score": null ,
"_source" : {
"policytype": "policy" ,
"@timestamp": "2019-10-27T12:17:28.010Z" ,
"fgt_type": "traffic" ,
"type": "fortigate" ,
"rcvdbyte": "226529" ,
"message": "Oct 27 12:17:27 10.120.120.1 date=2019-10-27,time=08: 17:27,devname="Bell-DC-Forti501E-Firewall",devid="FG5H1E5818903643",logid="0000000013",type="traffic",subtype="forward",level="notice",vd="root",eventtime=1572178647,srcip=x,srcport=64407,srcintf="ssl.root",srcintfrole="undefined",dstip=x,dstport=5601,dstintf="port9",dstintfrole="lan",poluuid="cf8a78dc-ce38-51e8-f16c-30a888cd6da6",sessionid=2190243146,proto=6,action="close",user="lthai",group="UDN-Montreal-Radius",authserver="Radius server",policyid=3,policytype="policy",service="tcp/5601",dstcountry="Reserved",srccountry="Reserved",trandisp="snat",transip=x,transport=64407,duration=126,sentbyte=16251,rcvdbyte=226529,sentpkt=148,rcvdpkt=185,appcat="unscanned",sentdelta=0,rcvddelta=0,dstdevtype="Router/NAT Device",dstdevcategory="None",masterdstmac="2c:23:3a:35:61:22",dstmac="2c:23:3a:35:61:22",dstserver=0" ,
"received_at": "2019-10-27T12:17:28.010Z" ,
"received_from": "{"name":"d0940d72625b","hostname":"d0940d72625b","os":{"name":"CentOS Linux","family":"redhat","version":"7 (Core)","platform":"centos","kernel":"4.4.0-166-generic","codename":"Core"},"containerized":true,"architecture":"x86_64"}" ,
"srcport": "64407" ,
"log" : {
"offset": 123797723 ,
"file" : {
"path": "/mnt/log/syslog"
}
},
"dstport": "5601" ,
"srccountry": "Reserved" ,
"syslog_timestamp": "Oct 27 12:17:27" ,
"srcip": "x" ,
"proto": "6" ,
"srcintf": "ssl.root" ,
"fgt_user": "lthai" ,
"authserver": "Radius server" ,
"level": "notice" ,
"trandisp": "snat" ,
"dstintfrole": "lan" ,
"appcat": "unscanned" ,
"dstmac": "2c:23:3a:35:61:22" ,
"syslog_message": "17:27,devname="Bell-DC-Forti501E-Firewall",devid="FG5H1E5818903643",logid="0000000013",type="traffic",subtype="forward",level="notice",vd="root",eventtime=1572178647,srcip=x,srcport=64407,srcintf="ssl.root",srcintfrole="undefined",dstip=x,dstport=5601,dstintf="port9",dstintfrole="lan",poluuid="cf8a78dc-ce38-51e8-f16c-30a888cd6da6",sessionid=2190243146,proto=6,action="close",user="lthai",group="yyy",authserver="Radius server",policyid=3,policytype="policy",service="tcp/5601",dstcountry="Reserved",srccountry="Reserved",trandisp="snat",transip=x,transport=64407,duration=126,sentbyte=16251,rcvdbyte=226529,sentpkt=148,rcvdpkt=185,appcat="unscanned",sentdelta=0,rcvddelta=0,dstdevtype="Router/NAT Device",dstdevcategory="None",masterdstmac="2c:23:3a:35:61:22",dstmac="2c:23:3a:35:61:22",dstserver=0" ,
"@version": "1" ,
"transip": "x" ,
"policyid": "3" ,
"dstdevcategory": "None" ,
"dstserver": "0" ,
"dstdevtype": "Router/NAT Device" ,
"dstintf": "port9" ,
"tags" : [
"fortigate_fb" ,
"beats_input_codec_plain_applied" ,
"_geoip_lookup_failure"
],
"geoip" : {},
"agent" : {
"hostname": "d0940d72625b" ,
"type": "filebeat" ,
"ephemeral_id": "3b86d83f-d3b4-4f28-8b63-7a2454732121" ,
"id": "bdd69604-f34d-4d04-86ec-7e8721d2bf9e" ,
"version": "7.4.0"
},
"ecs" : {
"version": "1.1.0"
},
"input" : {
"type": "log"
},
"action": "close" ,
"sentbyte": "16251" ,
"fgt_group": "UDN-Montreal-Radius" ,
"host" : {
"hostname": "d0940d72625b" ,
"name": "d0940d72625b" ,
"os" : {
"platform": "centos" ,
"kernel": "4.4.0-166-generic" ,
"family": "redhat" ,
"codename": "Core" ,
"version": "7 (Core)" ,
"name": "CentOS Linux"
},
"containerized": true ,
"architecture": "x86_64"
},
"syslog_hostname": "x" ,
"dstip": "x" ,
"dstcountry": "Reserved" ,
"srcintfrole": "undefined" ,
"transport": "64407" ,
"duration": "126" ,
"syslog_program": "date=2019-10-27,time=08" ,
"sentdelta": "0" ,
"rcvddelta": "0" ,
"fgt_subtype": "forward" ,
"fgt_service": "tcp/5601" ,
"masterdstmac": "2c:23:3a:35:61:22"
},
"fields" : {
"received_at" : [
"2019-10-27T12:17:28.010Z"
],
"@timestamp" : [
"2019-10-27T12:17:28.010Z"
]
},
"sort" : [
1572178648010
]
}

/>

Thanks

[Christian_Dahlqvist]

Have you refreshed the index pattern in Kibana?

[Long_Thai]

Yes I did and those fields that I mentioned remained non-searchable ?

[Long_Thai]

I forgot to put this block after the grok
<
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
/>

So everything is working now.

     grok {
       match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
   }

Thanks

[mandakh (meme)]

ISSUE>>
I have error with following script. Because of Elasticsearch 7 mapping types deprecated.
Could you guys help me to convert script to work on ES7

curl -X PUT "127.0.0.1:9200/_template/template_forti" -H 'Content-Type: application/json' -d'
{
"version" : 50002,
"order" : 1,
"template" : "fortinet-",
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"default" : {
"_all" : {
"enabled" : true,
"norms" : false
},
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
},
{
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date",
"include_in_all" : false
},
"@version" : {
"type" : "keyword",
"include_in_all" : false
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
},
"geodstip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
},
"geosrcip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo_point"
},
"latitude" : {
"type" : "half_float"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
},
"aliases" : { }
}
'

SOURCE>>

Fortigate Firewall ELK Configuration