Logstash to Elasticsearch connection

Elastic Stack Logstash
1

Hello!
I have filebeat running on 192.168.035 and ELK on 192.168.0.36 (name of the remote server is not good - " logstash" which can be confusing)
I'm runnnig filebeat -e -c filebeat.yml -d "publish" using logstash-tutorial-dataset file provided on filebeat Docs and everything looks good to me. Messages are sent to logstash and when I run tcpdump on port 5044 on ELK server I can see events are coming .
My filebeat.yml looks like:

filebeat.inputs:
- type: log
  id: my-filestream-id
  enabled: true
  paths:
    - /root/logstash-tutorial.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:
  host: "http://192.168.0.36:5601"    #my changes

output.logstash:
  hosts: ["192.168.0.36:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

my logstash-sample.conf

input {
  beats {
    port => 5044
  }
}

filter {
  dissect {
    mapping => {
      "message" => '"%{field1}" "%{field2}" "%{field3}" "%{field4}"'
    }    
  }
}

output {
  elasticsearch {
    hosts => ["192.168.0.36:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "-ffJv_8gjHgEpButf=Fq"
    }
    stdout {codec => rubydebug }
}

and my elasticsearch.conf

cluster.name: my-application
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0

xpack.security.enabled: false  #my
xpack.security.enrollment.enabled: false   #my
xpack.security.http.ssl:
  enabled: false   #my
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["logstash"]
http.host: 0.0.0.0

when I run from /usr/share/logstash/bin/
./logstash -f /etc/logstash/conf.d/logstash-sample.conf --config.test_and_exit
it says - my configuration is OK

questions:
1)
I stop logstash and run ./logstash -f /etc/logstash/conf.d/logstash-sample.conf --config.reload.automatic
the last lines of the output are:

INFO ] 2023-02-09 13:43:51.639 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.35}
[INFO ] 2023-02-09 13:43:51.695 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2023-02-09 13:43:51.752 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-02-09 13:43:51.854 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2023-02-09 13:43:51.909 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

I expected to see here messages are coming from filebeat - but nothing in output
after the last line. Should be like that?

I want to configure the most basic connection between logstash and elasticsearch and kibana -> based on username and password. No SSL, nothing.
Is my current logstash and elasticsearch configs are good for this goal?
3)
I can't see elasticsearch.log file in /var/log/elasticsearch directory, only my-application.log .. looks like my-application comes from cluster name.. but I believe there should be elasticsearch.log or I'm wtong?

Thank you

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.