Unable to parse multiple format logs


(Saketh Chandra Kolisetty) #1
here is my full configuration:

input {
file {
type => "apache"
path => "C:/Users/Mahe/Desktop/intership/apache.log"
}
file {
type => "csvf"
path => "C:/data/cars.csv"
}
}
filter {
if [type] == "apache" {
grok {
match => [
"message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
"message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
]
overwrite => [ "message" ]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
}
geoip {
source => "clientip"
target => "geoip"
add_tag => [ "apache-geoip" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
useragent {
source => "agent"
}
}
if [type] == "csvf" {
csv{
separator => ","
columns=>["maker","model","mileage","manufacture_year","engine_displacement","engine_power","body_type","color_slug","stk_year","transmission","door_count","seat_count","fuel_type","date_created","date_last_seen","price_eur"]
}
mutate{convert=>["mileage","integer"]}
mutate{convert=>["price_eur","float"]}
mutate{convert=>["engine_power","integer"]}
mutate{convert=>["door_count","integer"]}
mutate{convert=>["seat_count","integer"]}
}
}
output {
if [type] == "apache" {
elasticsearch {
hosts => "localhost"
index=>"apa"
document_type=>"apacsvd"
}
stdout{}
}
if [type] == "csvf" {
elasticsearch {
hosts => "localhost"
index=>"csvf"
document_type=>"apacsvd"
}
stdout{}

}

}
however when i try to parse them individually it works just fine


(Magnus Bäck) #2

Okay, but in what way is it not working now? What is the symptom?


(Saketh Chandra Kolisetty) #3

the pipeline gets started but logs are not parsed.


(Magnus Bäck) #4

How are you starting Logstash when it's not working? How are you starting Logstash when it is working? Have you tried bumping up Logstash's loglevel and looked in its log for clues?


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.