Help with config file for given syntax

Elastic Stack Logstash
1

i have a file with data in the format as given below, i have tried several times to index it using logstash conf file but failed. Please help me how can i index in two cases:
1: all key value pairs
2: some key value pairs.

-------------------------------
a=b
c=d
e=3
EOE
-------------------------------
a=z
c=l
e=4
EOE

2

You will want to start with a file input that uses the multiline codec.
Try specifying pattern as EOE, and what as previous.

Use the kv filter plugin to extract the key-value pairs. You might need to use two kv filters, the first splitting your multiline on the linebreak, and the second one then reading the individual fields to split the actual key-value pairs.

Test this with a simple stdout output.

3

I have tried doing something like this, but i think those "dash" lines aren't taken care of. Those dash lines are also in the input. is there any flaw in the code?

input {
file {
path => "/usr/local/Cellar/logstash/6.6.1/libexec/config/apache-daily-access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^EOE"
# negate = "true"
what => "previous"
}
}
}

filter {
mutate {
gsub => [
"message", "\n", ";" ] }
kv {
allow_duplicate_values => false
field_split => ";"
value_split => "="
include_keys => ["a", "c", "e" ]
}

mutate {
convert => [
"f", "integer"
]
}

}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
# region => "us-east-1"
index => "service_index"

}
stdout { codec => json }
}

4

how do i stop multiline filter for not including the dash lines and EOE in previous line?
this entire data is indexed as one line.
EOE
-------------------------------
a=z
c=l
e=4

5

You might need to set negate=true, and what=next.
This will group the events from ------ to EOE.

Then you would remove the ----- line and the EOE line from the message with a filter.

1 Like
7

thanks Benny it worked. i have 1 more question.
i have a file with kv pairs in which startTime key has output in two formats one is in epoch(1553438267.250) and other in unix time format(2019-03-24T14:37:48.096Z),
how do i convert epoch format to unix format during parsing and then how do i create @timestamp index on startTime values?

8

With the date filter you can actually match multiple patterns.

UNIX will match epoch date, and your other one looks like ISO8601 to me.

So you might want to use a date filter like this:

filter {
  date {
    match => ["startTime","UNIX","ISO8601"]
    target => "@timestamp"
  }
}

The default target is already @timestamp, so you can skip that line.

1 Like
9

i have posted another problem on different thread, please help me with that.

10

thanks benny. you rocks.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.