Cannot start filebeat with configuration file

Yasser_Alsawy · May 12, 2023, 9:05pm

I'm not able to start filebeat with below config file:

filebeat.inputs:
  - type: filestream
    id: input1-id
    paths:
      - /home/yasser/data/sample1.log
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: ["test-index"]

I get error:

{"log.level":"info","@timestamp":"2023-05-12T17:31:53.579+0300","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-05-12T17:31:53.580+0300","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: 56e8404f-da9e-403c-b3cb-225acdd386b8","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-05-12T17:31:53.580+0300","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: can not convert 'object' into 'string' accessing 'output.elasticsearch.index' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: **can not convert 'object' into 'string' accessing 'output.elasticsearch.index'** (source:'/etc/filebeat/filebeat.yml')
   ...fail!

stephenb · May 12, 2023, 9:16pm

Hi @Yasser_Alsawy Welcome to the community!

index is not an array
index: "test-index"

Yasser_Alsawy · May 13, 2023, 7:21am

Thanks it starts now but the index is not created:
curl -XGET localhost:9200/_cat/indices

Isn't this configuration enough to create the index?

filebeat.inputs:
  - type: filestream
    id: input1-id
    paths:
      - /home/yasser/data/sample1.log
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "test-index-%{+yyyy.MM.dd}]"
setup.template:
  name: "test-index-*"
  pattern: "test-index"

stephenb · May 13, 2023, 1:37pm

Is there data in the log file??

What version?

Filebeat will only try to load the data once so if it already tried it won't do it again

You will need to clean out the data registry.

What did the filebeat log show? Any errors?

Is elasticsearch really run you on HTTPS?

I suspect there's connection errors... You will need to supply a username and password.
If it's a self-signed shirt you're going to have to ignore the verification or provide the CA?

How did you set up elasticsearch?

If you want help, you're going to have to provide more detail.

Yasser_Alsawy · May 13, 2023, 8:10pm

it worked showing the index when I added below line to config file:

setup.ilm:
  enabled: false

now I can see the index at Discover but with no data. and it is still not shown at Index Management. and curl does not get this index as well. Does this mean index is created but not single transaction done so no single instance of this index exists?

the logs when starting show:

> 
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.161+0300","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.161+0300","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: 56e8404f-da9e-403c-b3cb-225acdd386b8","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.161+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"56e8404f-da9e-403c-b3cb-225acdd386b8"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.161+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"a8dbc6c06381f4fe33a5dc23906d63c04c9e2444","libbeat":"8.7.0","time":"2023-03-23T00:44:06.000Z","version":"8.7.0"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.161+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":20,"version":"go1.19.7"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.162+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1114},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-05-13T12:21:34+03:00","containerized":false,"name":"Yasser-Dell","ip":["127.0.0.1","::1","172.25.6.13","fe80::215:5dff:fed7:dc17"],"kernel_version":"5.15.90.1-microsoft-standard-WSL2","mac":["00:15:5d:d7:dc:17"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"+03","timezone_offset_sec":10800,"id":"3bda63e1e73958fadc5c99fa642befe5"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.162+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1143},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":2255,"ppid":2254,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2023-05-13T17:06:05.020+0300"},"ecs.version":"1.6.0"}}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.162+0300","log.origin":{"file.name":"instance/beat.go","file.line":297},"message":"Setup Beat: filebeat; Version: 8.7.0","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.166+0300","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://localhost:9200","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.166+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: Yasser-Dell","service.name":"filebeat","ecs.version":"1.6.0"}
> {"log.level":"info","@timestamp":"2023-05-13T17:06:05.166+0300","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
> Config OK

Yasser_Alsawy · May 13, 2023, 8:21pm

Elasticsearch runs properly at http localhost:9200
I neither installed a certificate nor set user/password. I have corrected input path to be http not https but still having same issue of no data uploaded.

the input file has just testing data in format:
name: value

I tried to load the file many times but data never uploaded, do I still to delete registry?

As you can see from logs, I run Elasticsearch and filebeat at my Windows 11 WSL (ubuntu) for testing. does this include any shortage or challenge?

stephenb · May 13, 2023, 10:17pm

Yes probably... Look here and clean the data directory

That should be fine...

One thing is I rarely advise starting with a custom index name if you are new as there are many benefits of using the defaults... But that is just a suggestion.

For new users I recommend start with the defaults, get to know the components, configuration and options and then start changing things...

If you had just taken the default filebeat.yml and simply enabled the filestream and put the path to your log file it would all probably already be working.

Yasser_Alsawy · May 14, 2023, 10:22am

I have followed your suggestion using default yml file with no index name:

###################### Filebeat Configuration Example #########################

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/sample1.log

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I ran all successfully before starting filebeat service:
./filebeat test config
./filebeat test output
./filebeat setup

but neither index exists at Kibana index management not kibana index patterns

but when running my customized yml with an index name, index pattern exists but no index at index management. I can find the index at Discover but with no data.

filebeat.inputs:
  - type: filestream
    id: input1-id
    paths:
      - /home/yasser/data/sample1.log
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "test-index-%{+yyyy.MM.dd}]"
setup.ilm:
  enabled: false
setup.template:
  enabled: false
  name: "test-index"
  pattern: "test-index-*"

setup.dashboards.index: "test-index-*"

how could I troubleshoot both cases?

Yasser_Alsawy · May 14, 2023, 10:32am

when trying to run filebeat using ./filebeat -e, instead of service filebeat start, as I used before, I got a clear error:

{"log.level":"info","@timestamp":"2023-05-14T13:26:56.036+0300","log.origin":{"file.name":"instance/beat.go","file.line":483},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-05-14T13:26:56.036+0300","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/bin/kibana: No directory /usr/share/filebeat/bin/kibana/7","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/bin/kibana: No directory /usr/share/filebeat/bin/kibana/7

should I set a dashboard property here as it is trying to read from a missing directory: /usr/share/filebeat/bin/kibana ?

stephenb · May 14, 2023, 1:31pm

With the default...

You need to set this to the location of your sample log.

/home/yasser/data/sample1.log
Also did you clean out the data directory as I instructed?

Then try starting as a service.

This will creat a data stream did you read about data stream...

Should be named

filebeat-8.7.1

Assuming you are using the latest version

Do not try to change anything with the dashboards, try running as a service again

Yasser_Alsawy · May 14, 2023, 4:28pm

I have moved my input log file into /var/log as the path dexcribed in template config file
I have deleted both log.json and meta.json from /var/lib/filebeat/registry/filebeat

but the link you provided shows data folder as:
data: The location for persistent data files. - {extract.path}/data
is this: /etc/filebeat ? I cannot find data folder there

I started filebeat as service (after restarting both elasticsearch and kibana)

Index management --> reload indices: nothing starting with filebeat is shown
Kibana --> Index patterns: only old index created by specifying the name is shown but nothing started with filebeat is shown
Discover: no new index or starting with filebeat is shown

###################### Filebeat Configuration Example #########################

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/sample1.log

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I'm running Elasticsearch 7.17.10 as shown from localhost:9200

stephenb · May 14, 2023, 4:39pm

From Kibana -> Dev Tools run

GET _cat/indices?v

Something simple is going on. If nothing else, just delete your entire installation of filebeat and start over. Not sure what went wrong.

And follow the quick start guide

I'm not an expert on the Ubuntu on Windows. I suppose it could be something to do with that...

But I don't think so because I think we'd get fundamental errors.... You're just having problems with reading data

Exactly. How did you install? Did you use a tar.gz or RPM

This makes a difference of how you run the setup command.

I think you were mixing things up and moving things around perhaps.

So exactly how did you install? What exact steps?

Yasser_Alsawy · May 14, 2023, 5:23pm

here is the indices for dev tools:

#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .geoip_databases                 DlDbDHzXRtqxnjRnOtrUgQ   1   0         37            0     35.6mb         35.6mb
yellow open   cars                             OGYpKAnQS0Wo-QSmglws3A   1   1         11            0     42.9kb         42.9kb
green  open   .apm-custom-link                 VeQQANccQSaqaakqTum_xg   1   0          0            0       226b           226b
green  open   .kibana_7.17.9_001               u_H7qUAZQYq20Aw5TD2wNQ   1   0        431           11      2.4mb          2.4mb
green  open   .apm-agent-configuration         sfhKwy0-QoKysQHszY_W4Q   1   0          0            0       226b           226b
green  open   .kibana_7.17.10_001              c07j9LEcQbejxNkFQyXKoA   1   0       2227           34      3.2mb          3.2mb
green  open   .kibana_task_manager_7.17.9_001  MzvCefHnRA2B_0n1kqRLSQ   1   0         17           16     19.5kb         19.5kb
green  open   .async-search                    hbo_Ww-wQBatJXSR1R5qvQ   1   0          0            0       256b           256b
green  open   .kibana_task_manager_7.17.10_001 lPWjIWv4R6GIn-cx6Zql0Q   1   0         17         3264      1.8mb          1.8mb
green  open   .tasks                           nDfvbnf5TkWgh0xSokV41A   1   0         27            0     53.2kb         53.2kb
yellow open   kids                             V_a-9pl-SFCCrugfd3TTqA   1   1          7            0     17.4kb         17.4kb

neither filebeat nor test-index I created using specific index name exists.
before going in uninstall and install again:

is any issue with v 7.17? how could I get v 8?
any issue with error for folder of kibana does not exist when starting using ./filebeat instead of starting as a service?

stephenb · May 14, 2023, 5:25pm

Exactly how did you install? ... the most important question to answer... which you have not.

Exactly
What did you download?
Exactly How did you download it?
Exactly What commands did you run?

I would uninstall / delete everything and reinstall, this should not be hard if you followed the exact steps in the quick start guide.

I would just use the tar.gz to start that is easiest to get to work.

THEN if that works you can make it a service.

This is not a 7.x vs 8.x issue, there is something fundamental incorrect.

Yasser_Alsawy · May 14, 2023, 6:07pm

When trying to uninstall elasticseardch I was getting error:
E: Conflicting values set for option Signed-By regarding source https://artifacts.elastic.co/packages/7.x/apt/ stable: /usr/share/keyrings/elastic.gpg !=
E: The list of sources could not be read.

so I did below (even I do not fully understand the keyrings):
yasser@Yasser-Dell:/usr/share/keyrings$ cd /etc/apt/sources.list.d
yasser@Yasser-Dell:/etc/apt/sources.list.d$ ls -ltr
total 12
-rw-r--r-- 1 root root 112 Apr 4 12:33 docker.list
-rw-r--r-- 1 root root 274 May 12 10:29 elastic-7.x.list
-rw-r--r-- 1 root root 112 May 14 20:54 elastic-8.x.list
yasser@Yasser-Dell:/etc/apt/sources.list.d$ sudo rm elastic-8.x.list
yasser@Yasser-Dell:/etc/apt/sources.list.d$ sudo rm elastic-7.x.list

it did not fix the problem of filebeat index, but I'm able to uninstall elasticsearch now and I'm able to run now:
sudo apt update

stephenb · May 14, 2023, 6:24pm

Apologies I was asking about how exactly you installed filebeat.

I am unclear on what the state of filebeat is at this point.

Yasser_Alsawy · May 14, 2023, 8:21pm

I remember I used only:
sudo apt install filebeat

should I use:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.6.2-x86_64.rpm

do you recommend uninstalling and installing again? keeping elasticsearch 7.17 as is?

stephenb · May 14, 2023, 8:39pm

Hi Yasser

I am trying to be very clear.... As I already recommended above

So yes uninstall, remove, delete, purge whatever you need to do ... clean up.

And then I would install with the tar.gz from the quick start guide I recommended above.

DO not change anything and Only enable the filestream and the path to your file

run

./filebeat setup -e
./filebeat -e

IF / When you get that working you can save that configuration cleanup and then install as an RPM if you like.

Again I suggest if you want to install with APY YUM .rpm or .deb that you follow the documentation here...

Again if you follow the quick start guide it tells you EXACTLY what to do .. .or you can just keep trying random commands

I would not recommend going to 8.x since your cluster is 7.x

Good luck I am sure you can get it working.

Yasser_Alsawy · May 15, 2023, 1:38pm

Thanks a lot, Stephenb for your support and patience!
I found the problem when I uninstalled every thing before installing again indifferent way:

I installed using Debian file (because I have ubuntu) which makes a real mess as some files are in /usr/share, /etc and the running directory as well
some parts of installation done with sudo which causes conflict at files access

I have:

uninstalled using: sudo apt remove
then removing the files from all folders
installing using only tar files so installation directory is at one place containing all running file, log and config file
I have used custom config file (sorry for not starting with the default template):

filebeat.inputs:
  - type: filestream
    id: input1-id
    paths:
      - /home/yasser/data/sample1.log
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "yasser-%{+yyyy.MM.dd}]"
setup.ilm:
  enabled: false
setup.template:
  enabled: false
  name: "yasser"
  pattern: "yasser-*"

but problem now, I find at Kibana index management:
yasser-2023.05.15] (I do not know why extra closing square bracket!)

while at discover only:
filebeat-*
with error message: No matching indices found: No indices match "filebeat-*"

and I cannot find kibana --> indext patterns
as I used to find at previous kibana

did I miss something here?

stephenb · May 15, 2023, 2:23pm

Typos.....it's right there

And since you create your own index you need to create your own index pattern it is not automatically created for you.

And shouldn't this be enabled

Also you might want to look at this post If you are indeed, set to on creating your own indices

Again, I normally suggest starting with the defaults and learning and then creating interesting things ...