Connect filebeat to ELK

Hello!

Could you provide link to connect filebeat to logstash ? jboss logs will be transmitted

Thank you

Are you trying to load the logs into Elasticsearch? if so you do not need logstash

Here are the same 2 links I gave you

Plus I would start with the Filebeat quickstart here to send directly to Elasticsearch get that working and THEN send through logstash if you need to.

Hi Stephen!
I'm trying to connect to logstash
May be you are right.. let's start with

Hi!

I've installed elasticsearch and kibana on my VM and again I'm not able to insert token created in kibana web interface
I'm getting

and it is just hanging in web interface, reloading and reloading again and again

What I'm doing wrong if I follow exactly the steps provided on Docs?

Thank you

Vassiliy Vins

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2022-11-21T12:48:09.233-07:00","message":"Holding setup until preboot stage is completed.","log":{"level":"INFO","logger":"root"},"process":{"pid":4947},"trace":{"id":"bfa662041749ed0d08f3f1cc2670ee48"},"transaction":{"id":"50a201cdef28163d"}}

that's the list of command and the order I've run


java -version

yum install java-1.8.0-openjdk -y

#create repo for elasticsearch
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch


vim /etc/yum.repos.d/elasticsearch.repo

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md


sudo yum install --enablerepo=elasticsearch elasticsearch -y

systemctl daemon-reload
systemctl start elasticsearch
systemctl enable elasticsearch
systemctl status elasticsearch

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana  and save the token.. it is valid 30 minutes only

####################### installing kibana
########### Kibana

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

vim /etc/yum.repos.d/kibana.repo
[kibana-8.x]
name=Kibana repository for 8.x package
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install kibana -y

only change in kibana.yml
server.host: "0.0.0.0"

systemctl daemon-reload 
systemctl start kibana
systemctl enable kibana
systemctl status kibana

go to browser and open localhost:5601
it will ask for token - paste there

than run
/usr/share/kibana/bin/kibana-verification-code

it will create code - input in browser  DONE



/usr/share/kibana/bin  starting script

Finally, I've figured out - not all browsers with the default configuration are allowed to connect to kibana. At first I was using Firefox, Firefox incognito - didn't help, then I installed Chromium and I got the kibana web interface immediately
I believe it should be mentioned in ELK docs.. it takes a lot of time to find the problem

Hello!

I've made very simple configuration on my virtual machine -
one host with Elasticsearch + Kibana 192.168.0.55 and another
with filebeat 192.168.0.48
Applied Elasticsearch token to kibana - everything works
Elasticsearch.yml:

cluster.name: mycluster
node.name: elk
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0


xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["elk"]

http.host: 0.0.0.0

filebeat.yml

filebeat.inputs:
- type: log
  id: my-nginx-log
  enabled: true
  paths:
    - /var/log/nginx/access*.log
tags: ["back"]
fields:
  env: test 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
   hosts: ["192.168.0.55:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

question1 -will Elasticsearch accept all data from filebeat to port 9200 by default or some additional configuration required to allow traffic from filebeat to ELK?
question2 - how can I check that Elasticsearch receives data from filebeat?
Thank you

#1
That defines that filebeat will send data via the elasticsearch output and yes you should define the port.

#2 Do you have Kibana installed if so go to Kibana -> Discover to view the data.

Perhaps you should watch some of the getting started / quick start videos or the guides.

Example

I also noticed these look like you are using shipping nginx logs we have a module that will parse that / automatically create dashboards etc..

See here .... nginx is even the example we show

Thank you for detailed answer

Let's retuarn back to my question. Do I need some extra configuration to for fillebeat and Elasticsearch to exchange info? or

output.elasticsearch:
   hosts: ["192.168.0.55:9200"]

is enough to send data and see them later on in Kibana?

I'm asking because the Doc says

output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 
  ssl:
    enabled: true
    ca_trusted_fingerprint: "b9a10bbe64ee9826abeda6546fc988c8bf798b41957c33d05db736716513dc9c" 

is this configuration compulsory or just given as an example of configuration?

Ahhh thanks for the clarification.

It depends on how you setup elasticsearch...

There are many options

Since it looks like you have enabled security in elasticsearch... According to your elasticsearch.yml above then you need to use the second configuration.

output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 
  ssl:
    enabled: true
    ca_trusted_fingerprint: "b9a10bbe64ee9826abeda6546fc988c8bf798b41957c33d05db736716513dc9c"

You could use a less secure version

output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 
  ssl:
    verification_mode: none

Or if you have not setup the filebeat_internal.user

output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  username: "elastic"
  password: "PASSWORD" 
  ssl:
    verification_mode: none

The configuration options and descriptions are here

as soon as I insert these lines related to username, password and ssl

filebeat.inputs:
- type: log
  id: my-nginx-log
  enabled: true
  paths:
    - /var/log/nginx/access*.log
tags: ["back"]
fields:
  env: test 
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
   hosts: ["192.168.0.55:9200"]
  username: "elastic"
  password: "4WQ=+RqYuArNBl-acew7"
  ssl:
     verification_mode: none
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

filebeat is getting this error in systemctl status filebeat:

[root@centos72 filebeat]# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2022-11-27 09:06:58 MST; 4min 17s ago
     Docs: https://www.elastic.co/beats/filebeat
  Process: 8115 ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 8115 (code=exited, status=1/FAILURE)

Nov 27 09:06:58 centos72 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Nov 27 09:06:58 centos72 systemd[1]: Unit filebeat.service entered failed state.
Nov 27 09:06:58 centos72 systemd[1]: filebeat.service failed.
Nov 27 09:06:58 centos72 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Nov 27 09:06:58 centos72 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Nov 27 09:06:58 centos72 systemd[1]: start request repeated too quickly for filebeat.service
Nov 27 09:06:58 centos72 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Nov 27 09:06:58 centos72 systemd[1]: Unit filebeat.service entered failed state.
Nov 27 09:06:58 centos72 systemd[1]: filebeat.service failed.

Needs to be... https

Also careful with your indentations above... In yml details matter... Although in this case it probably doesn't matter. Your hosts is not indented properly.

Also to see the actual logs per the docs..

journalctl -u filebeat.service

finally I figured out what caused the error message and didn't allow to start filebeat:
they were lines-
tags: ["back"]
fields:
env: test

Commented them out and now it works

Now next step how can I see my logs in kibana interface?

should I create role and user in kibana interface first? between filebeat and elasticsearch I'm using username "elastic" and pasword created elasticsearch during installation process

Hmm though should be valid configurations perhaps the syntax was wrong.

I would just use the elastic user to get started.

Please open new threads with new specific questions.

should be indention ... will check after because I need the configuration

Thank you for your help .. will open new thread

1 Like

Tags and fields should work... Glad you got it working!