How can split discovery message field?

Elastic Stack Beats
1

Elasticsearch version 5.5.1
kibana version 5.5.1
Logstash version 5.5.1
Beats version 5.5.1

my message field:

message:[INFO ] Status => SENT | client : [IPDC] | cell : [1746710009] | message-delivery-time : [2017-08-07 09:46:27,807] | Operator: [ROBI]

I would like to split according to Status, Client, Operator and message-delivery-time.
I wish any expert will help me.
Thanks

2

You will have to use the Logstash grok filter or dissect filter to parse your message.

3

my logstash.conf

filter {
if [type] == “syslog” {
grok {
match => { “message” => “%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_clientip} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}” }
add_field => [ “received_at”, “%{@timestamp}” ]
add_field => [ “received_from”, “%{host}” ]
}
syslog_pri {}
mutate {
remove_field => [ “type”, “tags”, “input_type”, “@version”, “beat”, “offset”]
}
date {
match => [ “syslog_timestamp”, “MMM d HH:mm:ss”, “MMM dd HH:mm:ss” ]
}
}
}

my discovery logs format:

@timestamp:August 8th 2017, 13:45:54.424 host:vNTDACLSnTALK01 source:/home/local/group/nazdaq/logs/naztech.log message:[INFO ] Status => SENT | client : [MTB] | cell : [1746710009] | message-delivery-time : [2017-08-07 09:46:27,807] | Operator: [GP]

Available field:
@timestamp
message
host message
source

I need available field:
@timestamp
message
host message
source
status
client
operator

What can I do? Please anybody help me.
thanks

4

please properly format logs and configs using the </> button.

I don't see how your grok pattern can match the log in any way, as you don't use any of the keywords in your grok pattern. You can try with https://grokdebug.herokuapp.com.

grok is basically build up regular expressions. One has to escape [ and | for example. You can also try to match field by field when testing by adding .* to the end of your pattern. Some simple testing: @timestamp:%{DATA:ts} host:%{DATA:host} source:%{DATA:source} message:\[%{DATA:level}\] Status => %{DATA:what} \| client : \[%{DATA:client}\] \|.*

5

filter {
mutate {
remove_field => [ "type", "tags", "input_type", "@version", "beat", "offset"]
}
}

my discovery logs format:

@timestamp:August 8th 2017, 13:45:54.424 host:vNTDACLSnTALK01 source:/home/local/group/nazdaq/logs/naztech.log message:[INFO ] Status => SENT | client : [MTB] | cell : [1746710009] | message-delivery-time : [2017-08-07 09:46:27,807] | Operator: [GP]

Available field:
@timestamp
message
host message
source

I need available field:
@timestamp
message
host message
source
status
client
operator
message-delivery-time

how can I add last four field?

thanks

6

I uses the .* pattern at the end of my grok pattern to ignore the rest of the log message. Remove the .* pattern and add the correct patterns for the other fields like I've already shown.

I only used DATA in my grok pattern. DATA is just .*. That is raw unparsed text. as you have timestamps and numbers you might want to replace data with the appropriate grok pattern types. You are highly encouraged to use the debugger when improving your grok pattern.

7

Again I change logstash filter.conf

filter {
grok {
match => { "message" => "@timestamp:%{DATA:ts} host:%{DATA:host} source:%{DATA:source} operator:%{DATA:operator} message:[%{DATA:level}] status => %{DATA:what} | client : [%{DATA:client}] " }
}
mutate {
remove_field => [ "type", "tags", "input_type", "@version", "beat", "offset"]
}
}

discovery logs:
@timestamp: August 27th 2017, 15:33:50.603
t host: vNTDACLSnTALK01
t message: log-level : [INFO ], status : [FAIL], client : [IPDC], cell : [1746710009], message_delivery_time : [2017-08-14 09:46:27,807], operator: [ROBI]
t source: /home/local/group/nazdaq/logs/naztech.log

but don't show field:

status
client
operator
message-delivery-time

So I need help
Thanks

8

grok is build on regular expressions. Regular expressions are used to match the contents against the regular language describe by the regular expression. When parsing some semi-structured data one must obey order and can not have arbitrary nesting.

Your grok pattern uses | as separate, but your sample log uses ,. Also please format using the </> button, I have no idea how correct your event is (like number of lines, whitespace and others...). Not just the separate did change, but also the order of the fields. Maybe the kv filter is a better fit for the initial parsing.

If you have Logstash configuration/parsing/processing related questions, better use the Logstash forum. There you will find users with much more experience in Logstash then in the filebeat forum.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.