Logstash.conf not found for after installing logstash

(Sawyer Ford) #1

I have installed the logstash with the below link but I am not able to find logstash.conf in my system at the paths



The OS is Ubuntu 16.04

Logtash dont capture Logger.error logs using logback
(Kofi) #2

What's inside of /etc/logstash?

(Sawyer Ford) #3

I find below files

sawyer@algo-11:/etc/logstash$ ls
conf.d jvm.options log4j2.properties logstash.yml startup.options

sawyer@algo-11:/etc/logstash$ cd conf.d
sawyer@algo-11:/etc/logstash/conf.d$ ls -lrt
total 0

When I provide the below commands it gives

sawyer@algo-11:~$ sudo systemctl start logstash.service
sawyer@algo-11:~$ sudo initctl start logstash
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

I have also run the command before installation

nbhagwath@algo-11: export JAVA_HOME=/opt/jdk1.8.0_151

(Magnus Bäck) #4

Logstash doesn't ship with a file named logstash.conf. Are you conflating that file with logstash.yml (which should be found in /etc/logstash) or what makes you think logstash.conf should exist anywhere?

(Sawyer Ford) #5

Fine now after going through the tutorials I was able to find it. But now I have given the encoder, inside the logback.xml. Only the debug and error logs are turned into JSON format, and other info, trace or not.

(Kofi) #6

So you do have a conf.d, you just don't have anything in it. You need to
create a file in it that has an input, filter and output inside. Then try
running things

(Sawyer Ford) #7

Hi, I have created.

Can you please let me know how do I capture all the types of the Logger.(ERROR, DEBUG, TRACE, INFO, and WARN)

(Kofi) #8

try debug or trace

(Sawyer Ford) #9

where should I try and configure the same

(Magnus Bäck) #10

where should I try and configure the same

Does the log file produced by your (Java?) application contain messages of all loglevels? If not then this isn't a Logstash problem.

(Kofi) #11

/etc/logstash/logstash.yml. at the bottom

(Kofi) #13

.I don't understand what you're asking

(Sawyer Ford) #14

@seanziee: I have edited my question

(Sawyer Ford) #15

@magnusbaeck: please answer to this question. It is the four questions above that I am facing available in this thread

(Magnus Bäck) #17

please answer to this question. It is the four questions above that I am facing available in this thread

Then please answer this question: Does the log file produced by your (Java?) application contain messages of all loglevels?

(Sawyer Ford) #18

Yes it has all the log levels

(Magnus Bäck) #19

Unless you have configured Logstash to drop certain messages I find it hard to believe that some messages aren't being processed. What does your Logstash configuration look like? And have you tried creating a minimal example that reproduces the problem? Like a tiny file with just a few log entries and a stdout { codec => rubydebug } Logstash output that just dumps all events?

(Sawyer Ford) #20

Thanks for the acknowledgement,

I have created a file named logstash.conf inside /etc/logstash/conf.d/logstash.conf. The configuration is as below. But when with the below command. Even though I have added the line path.settings: /etc/logstash/conf.d/ inside logstash.conf I still get the same issue.

  **sawyer@ford11:/usr/share/logstash$ bin/logstash -f logstash.conf**
  WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or 
  /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
  Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using 
  default config which logs errors to the console
 [ERROR] 2018-01-29 11:27:50.728 [Ruby-0-Thread-1: 
  sourceloader - No configuration found in the configured sources.

   input { stdin { } } 
   filter {
      mutate {
          strip => "message"
    output { 
            stdout { } 

(Magnus Bäck) #21

There are several problems here.

  • You're running bin/logstash -f logstash.conf from /usr/share/logstash but logstash.conf is stored in /etc/logstash/conf.d so the file obviously won't be found.
  • path.settings isn't a valid Logstash setting (see https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html).
  • Even if path.settings were valid it should go in the settings file, logstash.yml, not a pipeline configuration file stored in /etc/logstash/conf.d.

(Sawyer Ford) #22

Please let me know where do I run
bin/logstash -f logstash.conf

when I run as below I get the error as
saywer@ford11:/etc/logstash/conf.d$ logstash -f logstash.conf
logstash: command not found