Parsing multiple file and adding service-name


#1

hey!

I am currently using logstash-6.0.0. I have a microservices architecture where I store all my logs in /var/log/mservice/ and ops infrastrucutre (kibana, elasticsearch, logstash, grafana, etc) in /var/log/ops/

I have the following script which is not picking up anything anymore

##################################
# 1) INPUTS
##################################

input {
 
  # Microservice Logs
  file {
    type => "json"
    tags => ["json"]
    path => [
      "/var/log/mservices/*.log",
      "/var/log/mservices/**/*.log"
    ]
  }

  # Elasticsearch Logs
  file {
    type => "json"
    tags => ["elasticsearch"]
    start_position => "beginning"
    path => [
      "/var/log/ops/elasticsearch/*.log"
    ]
  }
  
  # Logstash Logs
  file {
    type => "json"
    tags => ["logstash"]
    start_position => "beginning"
    path => [
      "/var/log/ops/logstash/*.log"
    ]
  }
}

##################################
# 2) FILTERS
##################################

filter {

  # Microservice filters
  if "json" in [tags] {

    json {
      source => "message"
    }

    mutate {
      uppercase => ["level"] 
    }
  }

  # Elasticsearch Filters
  if "elasticsearch" in [tags] {
    
    mutate {
      add_field => { "service-name" => "elasticsearch" }
    }
  }

  # logstash Filters
  if "logstash" in [tags] {
    
    mutate {
      add_field => { "service-name" => "logstash" }
    }
  }
}

##################################
# 3) OUTPUTS
##################################

output {

  elasticsearch {
    hosts => "address"
  }
}

I tried to filter by type, but then I started getting the complaint:

Rejecting mapping update to [json, elasticsearch] as the final mapping would have more than 1 type multiple types. I therefore tried to filter on tags but this doesn't work :cry:
Due to reasons outside of the scope of this post, I can't use beats either.

How can I achieve what I trying to achieve?


#2

I found my answer :tada:
I posted an update on SO: https://stackoverflow.com/questions/49058258/logstash-parsing-multiple-files-multiple-file-type-error/49059459#49059459


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.