Logstash - Filebeat - Input not listening

Elastic Stack Logstash
1

Hi,

I am using 6.5.4 version of elastic stack. (Filebeat - Logstash - Elasticsearch - Kibana).

filebeat was working fine. Enabled x-pack in ES and other related settings.

ELK - Able to start the services.

But Logstash is not listening on port 5044. Filebeat input.

No error message Logstash and in Elasticsearch.

Logstash config file.

input {
  beats {
    port => 5044
  }
}
filter {
  mutate {
    copy => {
     "[fields][log_prefix]" => "[@metadata][log_prefix]"
     "[fields][log_idx]" => "[@metadata][index]"
     "[fields][application]" => "[@metadata][application]"
    }
  }
}
output {
  elasticsearch {
    user => logstash_internal
    password => x-pack-test-password
    hosts => ["HOSTNAME:9200"]
    manage_template => false
    index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

Logstash Log

[2019-02-25T07:00:32,940][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
[2019-02-25T07:00:33,301][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_internal:xxxxxx@HOSTNAME:9200/]}}
[2019-02-25T07:00:33,490][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://logstash_internal:xxxxxx@HOSTNAME:9200/"}
[2019-02-25T07:00:33,501][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-02-25T07:00:33,502][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-02-25T07:00:33,743][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://HOSTNAME:9200"]}
[2019-02-25T07:00:34,281][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x773be260 sleep>"}
[2019-02-25T07:00:34,413][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[:main]}
[2019-02-25T07:00:41,007][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

netstat -tunl | grep 5044 ID@HOSTNAME(PQ_Agility_1-7.5):/etc/logstash/conf.d 1

Can anyone help me on this?

2

The main pipeline is not running. How are you starting logstash? Are you using pipelines.yml?

3

Hi @Badger

Thanks for checking this.

How are you starting logstash?

sudo systemctl start logstash

Are you using pipelines.yml?

No. I dont have pipelines.yml. created pipeline in kibana with "main".

Logstash.yml

$ cat logstash.yml
path.data: /datavg/logstash/data
path.logs: /datavg/logstash/log
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: "x-pack-test-password"
xpack.monitoring.elasticsearch.url: "http://HOSTNAME:9200"
xpack.management.enabled: true
xpack.management.pipeline.id: ["main"]
xpack.management.elasticsearch.username: logstash_admin_user
xpack.management.elasticsearch.password: t0p.s3cr3t
xpack.management.elasticsearch.url: "http://HOSTNAME:9200"

I am new to Elastic Stack. Please correct me if I have missed any.

4

I do not use kibana, so I cannot assist with that.

6

Updated the pipeline with below content.

input {
  elasticsearch {
    user => logstash_admin_user
    password => "t0p.s3cr3t"
    hosts => ["HOSTNAME:9200"]
  }  
  beats {
    port => 5044
  }
}
filter {
  mutate {
    copy => {
     "[fields][log_prefix]" => "[@metadata][log_prefix]"
     "[fields][log_idx]" => "[@metadata][index]"
     "[fields][application]" => "[@metadata][application]"
    }
  }
}
output {
  elasticsearch {
    user => logstash_admin_user
    password => "t0p.s3cr3t"
    hosts => ["HOSTNAME:9200"]
  }  
  stdout { codec => rubydebug }
}
  • Logstash started listening 5044 port

$ netstat -atunl | grep 5044
tcp 0 0 0.0.0.0:5044 0.0.0.0:* LISTEN
$

But I am getting other exception.

[2019-02-26T11:16:19,450][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"http://ceala10649.emea.zurich.corp:9200/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s"}`

Error code 403 is auth error. But I have used admin user.

get _xpack/security/user/logstash_admin_user

{
"logstash_admin_user" : {
"username" : "logstash_admin_user",
"roles" : [
"logstash_reader",
"logstash_writer",
"logstash_admin"
],
"full_name" : "Logstash Admin User",
"email" : null,
"metadata" : { },
"enabled" : true
}
}

Can anyone tell what am I missing here?

7

I am using 6.5.4 version. Added "manage" permission to logstash_internal user and it is working fine.