vyjayanth
(Vyjayanth)
February 9, 2024, 9:57pm
1
Looking to drop a field called: Event.Original using drop_field. As Message Field produces same information as Event.Original.
I worked with remove_field of logstash filter, but it isn’t reflecting by dropping the field. Although it is nested representation of field, I was able to mention it as [event][original]. But it is not dropping the field.
Filebeat setting:
processors:
- decode_json_fields:
fields: ['event.original']
process_array: false
max_depth: 1
target: ''
overwrite_keys: false
add_error_key: true
- drop_fields:
fields: ['event.original']
logstash Filter setting: Inside Grok
remove_field => ["[event][original]"]
However, this setting is not dropping Field as required. Does anyone have suggestions around it?
Rios
(Rios)
February 10, 2024, 7:25am
2
Welcome to the community!
Just put: fields: ['event']
1 Like
vyjayanth
(Vyjayanth)
February 10, 2024, 9:36pm
3
Hey, thanks.. I was looking to drop ‘original’field nested in side “event”… does it reflect…
stephenb
(Stephen Brown)
February 10, 2024, 10:07pm
4
Hi @vyjayanth
Welcome to the community
So you architecture is
Filebeat > Logstash> Elasticsearch?
What version?
Typically event.original
is not created from filebeat It gets created by logstash or when the message arrives and goes through elasticsearch
That Would explain why you can't drop it at filebeat?
So tell us your architecture and the exact versions and share your logstash.
And then read about logstash ECS compatibility mode.
1 Like
vyjayanth
(Vyjayanth)
February 10, 2024, 10:35pm
5
Hi Stephen,
Yes, sorry that I didn’t mention before in the post.
We have a scenarios of Filebeat > Logstash > Splunk.
Filebeat : 8.8.0
Logstash: 8.8.2
Running as a side car container in rancher - k8.pod.
Oh ok, I was able to drop message field when I use drop-field feature in filebeat. Then I thought I could drop event.original.
I’ll bring in logstash conf for pipeline in a bit, not working today but outside…
stephenb
(Stephen Brown)
February 11, 2024, 1:01am
6
You can only drop fields that are available at the time the are shipped from filebeat... event.original
is most likely being created in logstash... not filebeat.
message
field is the main field from filebeat, that is why you could drop it
2 Likes
Rios
(Rios)
February 11, 2024, 4:04am
8
If you need to remove in LS:
mutate{ remove_field => [ "event" ] }
Also if you disable ECS on the pipeline(pipelines.yml or for all ,confs in logstash.yml), you will not see that field.
pipeline.ecs_compatibility: disabled
1 Like
vyjayanth
(Vyjayanth)
February 11, 2024, 10:24am
9
Hi Stephen and @Rios
I'm using the logstash setup for Springboot apps, here is the logstash's Filter Block. Please suggest any update, I'm new to GROK..
input {
beats {
port => 5044
}
}
filter {
if [fields][log-source] == "logfile" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time}:%{WORD:service} %{LOGLEVEL:loglevel} %{DATA:thread_name} \[%{DATA:src_file}:%{NUMBER:line}\] - %{GREEDYDATA:msg}" }
}
mutate {
add_field => { "tmp_path" => "%{[log][file][path]}" }
}
mutate {
split => ["tmp_path", "#"]
add_field => { "kubernetes.node.name" => "%{tmp_path[2]}" }
add_field => { "kubernetes.namespace" => "%{tmp_path[3]}" }
add_field => { "tmp_podname" => "%{tmp_path[4]}" }
}
mutate {
split => ["tmp_podname", "."]
add_field => { "kubernetes.pod.name" => "%{tmp_podname[0]}" }
}
mutate {
remove_field => [ "[event][original]", "tmp_path", "tmp_podname"]
}
}
}
output {
stdout {
codec => rubydebug
}
http {
format => "json"
content_type => "application/json"
http_method => "post"
url => "https://splunk-forwarder"
headers => ['Authorization', 'HEC']
ssl_verification_mode => none
}
}
I initially tried to mention in Mutate { }
, but with mutate{ remove_field => [ "[event][original]" ] }
it made no difference. I just tried out mentioning "event" in mutate filter.
mutate {
remove_field => [ "event", "tmp_path", "tmp_podname"]
}
It didn't drop the field.
Rios
(Rios)
February 11, 2024, 11:08am
10
Can you put a message sample?
vyjayanth
(Vyjayanth)
February 11, 2024, 11:39am
11
Hi @stephenb @Rios , it looks like disabling ecs has worked pretty well., it has dropped event{}.
Here is the event, after disabling ecs::
message: 2024-02-11 11:15:00,008 Application logger level info from .log file
input:
type: log
ENV: DEV
ecs:
version: 8.0.0
log:
offset: 4331
file:
path: "/app-logs/schapp.log"
tags:
- beats_input_codec_plain_applied
agent:
version: 8.8.0
id: f8c6981e-6221-49fb-8b83-64c5a5e5a0e4
name: podname-5d67c6789d-wz2v6
type: filebeat
ephemeral_id: 1234.1234.2314.5435.34534
"@version": '1'
NAMESPACE: <k8.namespacename>
host:
name: podname-5d67c6789d-wz2v6
os:
platform: ubuntu
name: Ubuntu
kernel: 3.10.0-1160.105.1.el7.x86_64
version: 20.04.6 LTS (Focal Fossa)
type: linux
codename: focal
family: debian
containerized: true
hostname: <hostname>
mac:
- aa-ff-gg-hgh-hh
ip:
- xx.xx.xx.xx
architecture: x86_64
"@timestamp": '2024-02-11T11:15:00.648Z'
p.s: changed the values fields
Rios
(Rios)
February 11, 2024, 1:56pm
12
Just for test, move mutatete remove_field out of IF. It can for sure remove any field.
filter {
if [fields][log-source] == "logfile"
bla bla...
} # end if
mutate {
remove_field => [ "event", "tmp_path", "tmp_podname"]
}
} # filter end
Or use the prune filter:
prune {
blacklist_names => [ "^event$", "^tmp_" ]
}
The parameter "pipeline.ecs_compatibility: disabled" doesn't use ECS however can be used as a workaround.
1 Like
system
(system)
Closed
March 10, 2024, 3:56pm
13
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.