Logstash config

Hi everyone,
I am newbie to elk stack so it would be so kind if you guy could help me.

I am able to send log directly to elasticsearch, but when I try to use logstash, i got below error when starting:

#logstash --path.settings /etc/logstash/
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-08-27T16:34:15,068][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-08-27T16:34:16,832][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 18, column 20 (byte 325) after output {\n elasticsearch {\n hosts => ["http://192.168.67.28:9200"]\n user => elastic\n password => CMC", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}
[2019-08-27T16:34:17,317][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-08-27T16:34:22,221][INFO ][logstash.runner ] Logstash shut down.

Below is my pipeline config:
input {
beats {
port => 5044
}
}

filter {
grok {
    match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}.log" }
}
date { match => ["ts", "MMM dd HH:mm:ss.SSS" ] remove_field => [ "ts" ]}
}

output {
  elasticsearch {
hosts => ["http://192.168.67.28:9200"]
user => 
password => 
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

Please help to guide me with this issue.

Your user and password options should be surrounded by double quotes.

Thank you for your response.

After added double quotes, i dont see the above error anymore.
However, i am facing new error:

[root@elk bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/logstash.yml
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-08-28T14:37:54,698][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-28T14:37:54,727][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.
[2019-08-28T14:37:54,743][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

It seem strange because i have only one instance of logstash kibana and elasticsearch, all deploy in single server

When I run ps -ef | grep logstash i got below result:

logstash 11822     1 99 14:44 ?        00:00:06 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
root     11859 11518  0 14:44 pts/0    00:00:00 grep --color=auto logstash

Does this mean i have 2 logstash process?

Please suggest the next step.

Yes, you already have one instance running and it prevents you starting a new one. Kill the old process and start a new one.

Hi,

I have killed the logstash process successfully and i can see logstash service running as normal.
However, i am not able to see logstash indicies in Index Management.

Can you guy help to provide a way to test the output from logstash.
Below is my current configuration file:

input {
  beats {
    port => 5044
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["192.168.67.28:9200"]
    user => "user"
    password => "password"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Also, a little question: From all other topic i have searched from Elastic Discuss, i can see people dont have to put their username and password in configuration file. So how can i achive the same?

THank you all!

However, i am not able to see logstash indicies in Index Management.

Impossible to comment without seeing both Elasticsearch and Logstash log.

Also, a little question: From all other topic i have searched from Elastic Discuss, i can see people dont have to put their username and password in configuration file. So how can i achive the same?

You can use at least environment variables or keystore to store credentials. Read documentation for more information. But probably people don't use security at all if you don't see username or password parameters in configuration file :slight_smile:

Hi admlko,

I have gone through the logstash log file and seem like it still encounter the issue:

[2019-09-03T11:53:47,635][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-09-03T11:53:50,004][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 30, column 7 (byte 739) after output {\n#  elasticsearch {\n#    hosts => [\"192.168.67.28:9200\"]\n#    user => \"elastic\"\n#    password => \"CMC@global123\"\n#    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n#    document_type => \"%{[@metadata][type]}\"\n#  }\nstdout {}\ncodec ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-09-03T11:53:50,379][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-09-03T11:53:55,491][INFO ][logstash.runner          ] Logstash shut down.

Can you guide me which way should i can continue to dig in? I seem so lost right now T.T

Hi all,

Thank you for your help.
I just delete the port number in hosts => ["192.168.67.28:9200"] to hosts => ["192.168.67.28"] and it work :smiley:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.