Unable to differentiate logs

Elastic Stack Elasticsearch
1

Hi All,

I have configured ELK in my server and able to see the logs in Kibana but i am unable to find out the environment wise logs in Kibana.I have different environments like DEV,INT,UAT and PROD. All environments has the logs.

how can be the logs be differentiated. For example one log entry is from DEV, one log entry is from INT. How can be marked the logs that it is DEV or INT or UAT or PROD environment.

Regards
Raja

2

You need to attach a field in there that does that.

3

Hi Mark,

Thank you

can you please provide me some example config to refer.

Regards
Raja

4

That depends entirely on how you send data to ES.

5

I am not filtering anything as of now. I want the environment wise logs as it is from server to Kibana.

6

How do those logs get to ES?

7

I have logstash where i am doing the configuration to ship the logs to ES.My configuration looks like below. Here i have all environment logs under /etc/logs

input
{
file
{
path => "/etc/logs/*.log"
}
}
output
{
elasticsearch
{
hosts => "localhost:9200"
}
}

8

I imagine you have a logstash in each different environment.
Add a field with the value of the environment.
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-add_field

input
{
  file
  {
    path => "/etc/logs/*.log"
    add_field => {"environment": "production"}
  }
}
output
{
  elasticsearch
  {
    hosts => "localhost:9200"
  }
}
9

Hi Raja,

it looks like you just have one server and collect logs from only one location...

Personally I would maybe use Filebeat to read the log files although you can do it with Logstash as well.

You will probably have to define more than one path of you have all logs in the same location, something like

input
{
  file
  {
    path => "/etc/logs/*.prod.log"
    add_field => {"environment": "production"}
  }
  file
  {
    path => "/etc/logs/*.dev.log"
    add_field => {"environment": "DEV"}
  }
  file
  {
    path => "/etc/logs/*.int.log"
    add_field => {"environment": "INT"}
  }
}

Adjust according to your naming convention of log files :slight_smile:

Cheers,
AB

10

Thanks Ab and Pablo

11

Hi Ab,

This is not working. I have done the similar way but its throwing an invalid configuration.

12

Hi Raja,
sorry, I haven't used logstash this way... Looking at https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html looks like I left out the input type from the second and third path. I edited my previous post to fix that.

13

I fixed finally. I have created different configuration files referring same elasticsearch output so that i was able to make it possible.

Anyway Thank you