Incorrect field type

Elastic Stack Beats
1

Hi Team,
I am working something with (semi apache access log), copied below and i have also published my logstash conf file, i could see bytes,IP,date and respsonse code usually considerd as strings in kibana. I am able to resolve the problems with bytes and response by exclusively defining the mutate filter but the problems with IP and date persists.

0 10.00.00.208 - - [25/Jul/2018:00:00:08 -0700] "GET /Hollywood/movies/images/MT_searchMagnifier.svg HTTP/1.1" 200 388 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/MT_searchMagnifier.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

0 10.00.00.208 - - [25/Jul/2018:00:00:08 -0700] "GET /Hollywood/movies/images/MT_Mylist_normal.svg HTTP/1.1" 200 466 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/MT_MySales_normal.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

0 10.00.00.208 - - [25/Jul/2018:00:00:08 -0700] "GET /Hollywood/movies/images/MT_Openmovies_normal.svg HTTP/1.1" 200 442 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Win64; x
64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/MT_Openmovies_normal.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

0 10.00.00.208 - - [25/Jul/2018:00:00:08 -0700] "GET /Hollywood/movies/images/MT_Transactions_normal.svg HTTP/1.1" 200 371 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Win64; x
64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/MT_Transactions_normal.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

0 10.00.00.208 - - [25/Jul/2018:00:00:08 -0700] "GET /Hollywood/movies/images/MT_Dasboard_onClick.svg HTTP/1.1" 200 387 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/MT_Dasboard_onClick.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

0 10.00.00.208 - - [25/Jul/2018:00:00:08 -0700] "GET /Hollywood/movies/images/MT_Dasboard_myClients_normal.svg HTTP/1.1" 200 663 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Wi
n64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/MT_Dasboard_myClients_normal.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

0 10.00.00.208 - - [25/Jul/2018:00:00:09 -0700] "GET /Hollywood/movies/images/down_Arrow_Blue.svg HTTP/1.1" 200 239 "https://movies.ptin8.corp.prk.com/Hollywood/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" GET /Hollywood/movies/images/down_Arrow_Blue.svg "-" "" "HTTP/1.1" prkolnx5124 "-" "-"

My logstash conf file

input {
beats {
port => "5041"
}
}
filter {
grok {
match => { "message" => "%{CISCO_REASON:ignor1} %{COMBINEDAPACHELOG} %{GREEDYDATA:ignore2}" }
}
mutate {
convert => { "bytes" => "integer" }
}
mutate {
convert => { "response" => "integer" }
}
mutate {
strip => ["ignore1","ignore2"]
}

}

2

Additional information, i have tried date filter many times but still getting date parser exception in the logs.

3

What does the date field look like if you use

output { stdout { codec => rubydebug } }
4

My timestamp look like below.

"timestamp" => "25/Jul/2018:00:00:08 -0700",

5

How did you configure the date filter? Did you give it a pattern that looks like your date format string? It's a bit of a funky format, so you'll likely need to refer to the documentation.

6

Worth noting: the type of a field in Elasticsearch cannot be changed. Once the field is created in an index, that index will attempt to coerce anything that it is given for that field into the field's type, so even if you get Logstash to output a different type, the type of the field in Elasticsearch will not change.

7

ok, i will share the date pattern shortly.. Meanwhile, i am facing a new problem here with my filebeat.

I have taken a sample file containing exactly around 40 messages(format already mentioned in the earlier post)in them and started parsing it through filebeat, Although exclude filter and multiline pattern were set, i was expecting exactly 26 messages to be displayed in kibana,but i could only see 22 in them and missing the more important messages.

Filebeat yml:

  • type: log

    enabled: true

    paths:

    • /opt/bea/Logwarhouse/apache/access.log

    exclude_lines: ['a10hm/1.0']

    multiline.pattern: '^\d\s\d{2}'

    multiline.negate: false

    multiline.match: before

8

my date filter looks as below

mutate {
rename => { "timestamp" => "Apachetimestamp" }
}
date {
match => [ "Apachetimestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
}

9

when i give the above date filter, i could see logstash creating indexes for every single timeline availables in the message, which is very strange

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.