It also looks like you're using a default mapping, but you should create your own through an index template
Not sure why you have _ignored. message.keyword
So here is a mapping (you should complete it with all the fields from the message.
I used KV processor and cleaned up some stuff.
There is an issues and that is your message has an @timestamp
so you either put the message fields under a target_field
or you exclude the timestamp "exclude_keys" : ["@timestamp"]
.. your choice.... or you could add more logic to save to target fields replace the time stamp then copy everything to root... I like the target_field
approach
# Mapping
DELETE discuss-test-syslog
PUT discuss-test-syslog/
{
"mappings": {
"properties": {
"SOURCE": {
"type": "keyword"
},
"PROGRAM": {
"type": "keyword"
},
"PRIORITY": {
"type": "keyword"
},
"ISODATE": {
"type": "date"
},
"HOST_FROM": {
"type": "ip"
},
"HOST": {
"type": "ip"
},
"FACILITY": {
"type": "keyword"
},
"@timestamp": {
"type": "date"
},
"MESSAGE": {
"type": "keyword"
}
}
}
}
GET discuss-test-syslog/_search
DELETE discuss-test-syslog
PUT discuss-test-syslog/
{
"mappings": {
"properties": {
"SOURCE": {
"type": "keyword"
},
"PROGRAM": {
"type": "keyword"
},
"PRIORITY": {
"type": "keyword"
},
"ISODATE": {
"type": "date"
},
"HOST_FROM": {
"type": "ip"
},
"HOST": {
"type": "ip"
},
"FACILITY": {
"type": "keyword"
},
"@timestamp": {
"type": "date"
},
"MESSAGE": {
"type": "keyword"
}
}
}
}
# Ingest Pipeline
DELETE _ingest/pipeline/discuss-test-syslog
PUT _ingest/pipeline/discuss-test-syslog
{
"processors": [
{
"kv": {
"field": "MESSAGE",
"field_split": ",",
"value_split": ":",
"trim_key": "{\\\"",
"trim_value": "\\\"}",
"target_field": "details"
}
}
]
}
# Post with the pipeline
POST discuss-test-syslog/_doc?pipeline=discuss-test-syslog
{
"SOURCE": "s_network_fw",
"PROGRAM": "xxxxx",
"PRIORITY": "notice",
"MESSAGE": "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"RULE_DSC\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}",
"ISODATE": "2022-07-04T20:08:47+02:00",
"HOST_FROM": "192.168.1.1",
"HOST": "192.168.2.1",
"FACILITY": "user",
"@timestamp": "2022-07-04T20:08:47+02:00"
}
GET discuss-test-syslog/_search
Easy way to quickly iterate and test pipeline and simulate all in one.
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"kv": {
"field": "MESSAGE",
"field_split": ",",
"value_split": ":",
"trim_key": "{\\\"",
"trim_value": "\\\"}",
"exclude_keys" : ["@timestamp"]
}
}
]
},
"docs": [
{
"_source": {
"SOURCE": "s_network_fw",
"PROGRAM": "xxxxx",
"PRIORITY": "notice",
"MESSAGE": "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"RULE_DSC\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}",
"ISODATE": "2022-07-04T20:08:47+02:00",
"HOST_FROM": "192.168.1.1",
"HOST": "192.168.2.1",
"FACILITY": "user",
"@timestamp": "2022-07-04T20:08:47+02:00"
}
}
]
}