Logstash config

Elastic Stack Logstash
1

how can I config logstash to read two log files from local then pass them into two indies

2

Hi, without any more data try this to send into 2 indices

input {
  file {
	path => [ "/var/log/file1.log", "/var/log/file2.log" ]
  }
}
filter {
    ##to do filter data
}
output {
#output to file
#  file {
#    path => "/tmp/localData.log"
#  }
#output to elasticsearch
  if [log][file][path] == '/var/log/file2.log' {
    elasticsearch {
      hosts => ["192.168.x.xxx:9200"]
      manage_template => false
      index => "logstash-A-%{+YYYY.MM.dd}"
      ssl => true
      ssl_certificate_verification => false
      api_key => "xxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxxxxxxxxxxxxx"
    }
   else {
    elasticsearch {
      hosts => ["192.168.x.xxx:9200"]
      manage_template => false
      index => "logstash-B-%{+YYYY.MM.dd}"
      ssl => true
      ssl_certificate_verification => false
      api_key => "xxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxxxxxxxxxxxxx"
    }
  }
}
1 Like
3

Hi, thank you for your response. But it doesn't f\work for me. Here my config file

input {
  file {
    path => ["C:/nginx/logs/postcode/reverse-access.log", "C:/nginx/logs/behzisti/reverse-access.log"]
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => '%{IPORHOST:clientip} - %{USERNAME:username} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-)' }
  }
}

output {
  if [log][file][path] == "C:/nginx/logs/postcode/reverse-access.log" {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      index => "webservices_nginx"
      ssl_certificate_verification => false
    }
  } 
  else if [log][file][path] == "C:/nginx/logs/behzisti/reverse-access.log" {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      index => "behzisti_nginx"
      ssl_certificate_verification => false
    }
  }
}
4

Try instead

if [log][file][path] == "C:/nginx/logs/postcode/reverse-access.log" {

this

if [log][file][path] !~ /postcode/ {
  ....
} else if [log][file][path]  !~ /behzisti/ {
  ....
}

this means that if the log.file.path contains "postcode" do the firts if and if contains "behzisti" do the second if

5

I tested config for one by one and they worked. but doesn't work for two input together.
do I need some extra configuration for Elasticsearch?

6

Create 2 logstash configuration files one each one, and name it dinstincts each one.
This will work. Or you have centraliced pipelines?

7

actually, this is my main config file. I want analyze data from local logs and from beatfile.

input {
  file {
    path => ["C:/nginx/logs/postcode/reverse-access.log", "C:/nginx/logs/behzisti/reverse-access.log"]
    start_position => "beginning"
    type => "nginx"
  }
  beats {
    port => 5044
    type => "wso2"
  }
}

filter {
  if [type] == "nginx" {
    if [path] == "C:/nginx/logs/postcode/reverse-access.log" {
      mutate {
        add_field => { "index_name" => "webservices_nginx" }
      }
    } else if [path] == "C:/nginx/logs/behzisti/reverse-access.log" {
      mutate {
        add_field => { "index_name" => "behzisti_nginx" }
      }
    }
    grok {
      match => { "message" => '%{IPORHOST:clientip} - %{USERNAME:username} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-)' }
    }
  }
  else if [type] == "wso2" {
    grok {
        match => ["message", "%{GREEDYDATA:UNWANTED}\ apimMetrics:%{GREEDYDATA:apimMetrics}\, %{GREEDYDATA:UNWANTED} \:%{GREEDYDATA:properties}"]
    }
  }
}

output {
  if [type] == "nginx" {
    if [index_name] == "webservices_nginx" {
      elasticsearch {
        hosts => ["https://localhost:9200"]
        index => "webservices_nginx"
        ssl_certificate_verification => false
      }
    } else if [index_name] == "behzisti_nginx" {
      elasticsearch {
        hosts => ["https://localhost:9200"]
        index => "behzisti_nginx"
        ssl_certificate_verification => false
      }
    }
  } else if [type] == "wso2" {
    if [apimMetrics] == " apim:response" {
      elasticsearch {
        hosts => ["https://localhost:9200"]
        index => "apim_event_response"
        ssl_certificate_verification => false
      }
    } else if [apimMetrics] == " apim:faulty" {
      elasticsearch {
        hosts => ["https://localhost:9200"]
        index => "apim_event_faulty"
        ssl_certificate_verification => false
      }
    }
  }
}
8

Also I tried two config files separately, but just the frist config was apply.

9

You can have two config files for example 00-config.conf and 01-config.conf, when you restart your logstash service it must read all the configs,

I have in logstash multiple config files, i use 001-xxxxx.conf for the input data, 10-xxxxx.conf to filter the data and 30-output-xxxx.conf to configure the multiple indices output

10

I solved it :
first I defined three config file separately then I changed pipelines.yml :

- pipeline.id: nginx
  path.config: "C:\\Elastic\\logstash-8.7.0\\config\\logstash-sample1.conf"

- pipeline.id: nginx_behzisti
  path.config: "C:\\Elastic\\logstash-8.7.0\\config\\logstash-sample2.conf"

- pipeline.id: wso2
  path.config: "C:\\Elastic\\logstash-8.7.0\\config\\logstash-sample3.conf"
1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.