# Logstash config

**URL:** <https://discuss.elastic.co/t/logstash-config/196916>\
**Category:** Logstash\
**Created:** [August 27, 2019, 9:48am UTC](https://discuss.elastic.co/t/logstash-config/196916 "2019-08-27T09:48:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ntlong](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@ntlong](https://discuss.elastic.co/u/ntlong)\
**Post date:** [August 27, 2019, 9:48am UTC](https://discuss.elastic.co/t/logstash-config/196916/1 "2019-08-27T09:48:24Z")

</div>

Hi everyone,  
I am newbie to elk stack so it would be so kind if you guy could help me.

I am able to send log directly to elasticsearch, but when I try to use logstash, i got below error when starting:

> #logstash --path.settings /etc/logstash/  
> Thread.exclusive is deprecated, use Thread::Mutex  
> Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties  
> [2019-08-27T16:34:15,068][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.3.1"}  
> [2019-08-27T16:34:16,832][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, {, } at line 18, column 20 (byte 325) after output {\n elasticsearch {\n hosts =\> ["[http://192.168.67.28:9200](http://192.168.67.28:9200)"]\n user =\> elastic\n password =\> CMC", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]}  
> [2019-08-27T16:34:17,317][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
> [2019-08-27T16:34:22,221][INFO][logstash.runner] Logstash shut down.

Below is my pipeline config:  
input {  
beats {  
port =\> 5044  
}  
}

```
filter {
grok {
    match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}.log" }
}
date { match => ["ts", "MMM dd HH:mm:ss.SSS"] remove_field => ["ts"]}
}

output {
  elasticsearch {
hosts => ["http://192.168.67.28:9200"]
user => 
password => 
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

```

Please help to guide me with this issue.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 27, 2019, 12:39pm UTC](https://discuss.elastic.co/t/logstash-config/196916/2 "2019-08-27T12:39:03Z")

</div>

Your user and password options should be surrounded by double quotes.

---

<div class="post-metadata">

**Author:** ![ntlong](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@ntlong](https://discuss.elastic.co/u/ntlong)\
**Post date:** [August 28, 2019, 7:39am UTC](https://discuss.elastic.co/t/logstash-config/196916/3 "2019-08-28T07:39:57Z")

</div>

Thank you for your response.

After added double quotes, i dont see the above error anymore.  
However, i am facing new error:

```
[root@elk bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/logstash.yml
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-08-28T14:37:54,698][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-28T14:37:54,727][FATAL][logstash.runner] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2019-08-28T14:37:54,743][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

```

It seem strange because i have only one instance of logstash kibana and elasticsearch, all deploy in single server

When I run ps -ef | grep logstash i got below result:

```
logstash 11822 1 99 14:44 ? 00:00:06 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
root 11859 11518 0 14:44 pts/0 00:00:00 grep --color=auto logstash

```

Does this mean i have 2 logstash process?

Please suggest the next step.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [August 28, 2019, 8:10am UTC](https://discuss.elastic.co/t/logstash-config/196916/4 "2019-08-28T08:10:31Z")

</div>

Yes, you already have one instance running and it prevents you starting a new one. Kill the old process and start a new one.

---

<div class="post-metadata">

**Author:** ![ntlong](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@ntlong](https://discuss.elastic.co/u/ntlong)\
**Post date:** [August 30, 2019, 7:03am UTC](https://discuss.elastic.co/t/logstash-config/196916/5 "2019-08-30T07:03:02Z")

</div>

Hi,

I have killed the logstash process successfully and i can see logstash service running as normal.  
However, i am not able to see logstash indicies in Index Management.

Can you guy help to provide a way to test the output from logstash.  
Below is my current configuration file:

```
input {
  beats {
    port => 5044
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["192.168.67.28:9200"]
    user => "user"
    password => "password"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

Also, a little question: From all other topic i have searched from Elastic Discuss, i can see people dont have to put their username and password in configuration file. So how can i achive the same?

THank you all!

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [August 30, 2019, 7:19am UTC](https://discuss.elastic.co/t/logstash-config/196916/6 "2019-08-30T07:19:55Z")

</div>

> However, i am not able to see logstash indicies in Index Management.

Impossible to comment without seeing both Elasticsearch and Logstash log.

> Also, a little question: From all other topic i have searched from Elastic Discuss, i can see people dont have to put their username and password in configuration file. So how can i achive the same?

You can use at least environment variables or keystore to store credentials. Read documentation for more information. But probably people don't use security at all if you don't see username or password parameters in configuration file 🙂

---

<div class="post-metadata">

**Author:** ![ntlong](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@ntlong](https://discuss.elastic.co/u/ntlong)\
**Post date:** [September 3, 2019, 4:59am UTC](https://discuss.elastic.co/t/logstash-config/196916/7 "2019-09-03T04:59:01Z")

</div>

Hi admlko,

I have gone through the logstash log file and seem like it still encounter the issue:

```
[2019-09-03T11:53:47,635][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-09-03T11:53:50,004][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 30, column 7 (byte 739) after output {\n# elasticsearch {\n# hosts => [\"192.168.67.28:9200\"]\n# user => \"elastic\"\n# password => \"CMC@global123\"\n# index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n# document_type => \"%{[@metadata][type]}\"\n# }\nstdout {}\ncodec ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
[2019-09-03T11:53:50,379][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2019-09-03T11:53:55,491][INFO][logstash.runner] Logstash shut down.

```

Can you guide me which way should i can continue to dig in? I seem so lost right now T.T

---

<div class="post-metadata">

**Author:** ![ntlong](https://avatars.discourse-cdn.com/v4/letter/n/ce73a5/32.png) [@ntlong](https://discuss.elastic.co/u/ntlong)\
**Post date:** [September 3, 2019, 5:13am UTC](https://discuss.elastic.co/t/logstash-config/196916/8 "2019-09-03T05:13:36Z")

</div>

Hi all,

Thank you for your help.  
I just delete the port number in hosts =\> ["192.168.67.28:9200"] to hosts =\> ["192.168.67.28"] and it work 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 1, 2019, 5:13am UTC](https://discuss.elastic.co/t/logstash-config/196916/9 "2019-10-01T05:13:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.