Logstash fails on startup

Elastic Stack Logstash
1

So I've narrowed this issue down to this line in my example.conf (/etc/logstash/conf.d/example.conf)

input {
  tcp {
    port => 5002
    type => syslog
  }
  udp {
    port => 5002
    type => syslog
  }
  beats {
    port => 5044
    ssl => true
    ssl_key => '/etc/logstash/config/certs/logstash.pkcs8.key'
    ssl_certificate => '/etc/logstash/config/certs/logstash/logstash.crt'
    }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
       hosts => [ "https://node1.altignus.com:9200" ]
       cacert => '/etc/logstash/config/certs/ca/ca.crt'
       user => 'elastic'
       password => ################
       stdout { codec => rubydebug }
       #index => syslog-direct
       data_stream => "true"
       stdout
    }

The error I'm getting on startup is this:


[2021-09-26T09:46:16,110][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at line 30, column 14 (byte 752) after output {\n       hosts ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391:in `block in converge_state'"]}
[2021-09-26T09:46:16,638][INFO ][org.reflections.Reflections] Reflections took 204 ms to scan 1 urls, producing 120 keys and 417 values
[2021-09-26T09:46:18,075][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["https://node1.altignus.com:9200"]}
[2021-09-26T09:46:18,166][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@node1.altignus.com:9200/]}}
[2021-09-26T09:46:18,264][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@node1.altignus.com:9200/"}
[2021-09-26T09:46:18,281][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.15.0) {:es_version=>7}
[2021-09-26T09:46:18,283][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-09-26T09:46:18,469][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-09-26T09:46:18,469][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-09-26T09:46:18,506][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-09-26T09:46:18,717][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x3a295a30@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
[2021-09-26T09:46:20,245][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>1.52}
[2021-09-26T09:46:20,314][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2021-09-26T09:46:22,435][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2021-09-26T09:46:22,753][INFO ][logstash.runner          ] Logstash shut down.

Can somone point me in the correct direction here as when I comment out #stdout directive in example.conf it works but I have 0 messages loading from syslog in ELK.

2

This points exactly in your config file where the error is, in this case it is in line 30.

And looking at that line, your output is wrong, you are not saying which output plugin you are using.

Your output configuration should be something like this, with the elasticsearch output plugin and, if you want to keep it, the stdout output plugin.

output {
    elasticsearch {
       hosts => [ "https://node1.altignus.com:9200" ]
       cacert => '/etc/logstash/config/certs/ca/ca.crt'
       user => 'elastic'
       password => ################
       #index => syslog-direct
       data_stream => "true"
    }
    stdout { codec => rubydebug }
}
3

Thanks again;

I have updated the (/etc/logstash/conf.d/example.conf) to:

input {
  tcp {
    port => 5002
    type => syslog
  }
  udp {
    port => 5002
    type => syslog
  }
  beats {
    port => 5044
    ssl => true
    ssl_key => '/etc/logstash/config/certs/logstash.pkcs8.key'
    ssl_certificate => '/etc/logstash/config/certs/logstash/logstash.crt'
    }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
       elasticsearch {
                hosts => ["https://node1.altignus.com:9200"]
                cacert => '/etc/logstash/config/certs/ca/ca.crt'
                user => 'elastic'
                password => '####################'
                data_stream => "true"
                }
       stdout   {
                codec => rubydebug
                }
}

The error was 'elastisearch' instead of 'Elasticsearch' and the STDOUT location;


[2021-09-26T14:33:20,562][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>1.55}
[2021-09-26T14:33:20,671][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2021-09-26T14:33:21,023][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.88}
[2021-09-26T14:33:21,263][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-09-26T14:33:21,851][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-09-26T14:33:21,886][INFO ][logstash.inputs.tcp      ][main][d3e11169305a6e763c12c46e872e16bf5328e21900213c7d02d5583b0793c691] Starting tcp input listener {:address=>"0.0.0.0:5002", :ssl_enable=>false}
[2021-09-26T14:33:22,006][INFO ][org.logstash.beats.Server][main][f40ce36252b7bf234b33f4e971056ee96be8e6a4293bb8bd20e78d6eaf7db505] Starting server on port: 5044
[2021-09-26T14:33:22,040][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2021-09-26T14:33:22,107][INFO ][logstash.inputs.udp      ][main][56e9e02a25dd81d07a91f38db5db1f18ff0bd4b7d3b93dc8ba678baef1dd36be] Starting UDP listener {:address=>"0.0.0.0:5002"}
[2021-09-26T14:33:22,177][INFO ][logstash.inputs.udp      ][main][56e9e02a25dd81d07a91f38db5db1f18ff0bd4b7d3b93dc8ba678baef1dd36be] UDP listener started {:address=>"0.0.0.0:5002", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}

'''

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.