Filebeat output to file

Elastic Stack Beats
1

New to the filebeat and to elastic.
I need to fetch o365 logs from azure tenant.
I dont want to use ELK stack but just get the json files

I configured /etc/filebeat/modules.d/o365.yml

also file output under /etc/filebeat/filebeat.yml

================================== Outputs ===================================

Configure what output to use when sending the data collected by the beat.

-------------------------------- File Output ---------------------------------

output.file:

Boolean flag to enable or disable the output module.

enabled: true

Configure JSON encoding

codec.json:
# Pretty-print JSON event
pretty: true

# Configure escaping HTML symbols in strings.
#escape_html: false

I dont see any file output under "/tmp/filebeat"

I see this warning under /var/log/filebeat/filebeat-20220911-13.ndjson

{"log.level":"warn","@timestamp":"2022-09-11T15:58:24.406+0530","log.origin":{"file.name":"beater/filebeat.go","file.line":166},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}

What am I missing?

thanks for any help.

2

Set in filebeat.yml

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

If you want to save in a file, set :

output.file:
  path: "/tmp/filebeat"
  filename: filebeat
3

I have this already set:

# -------------------------------- File Output ---------------------------------
output.file:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Configure JSON encoding
  codec.json:
    # Pretty-print JSON event
    pretty: true

    # Configure escaping HTML symbols in strings.
    #escape_html: false

  # Path to the directory where to save the generated files. The option is
  # mandatory.
  path: "/tmp/filebeat"

  # Name of the generated files. The default is `filebeat` and it generates
  # files: `filebeat-{datetime}.ndjson`, `filebeat-{datetime}-1.ndjson`, etc.
  filename: filebeat
4

If you don't want Elasticsearch, comment:

# output.elasticsearch:
#  hosts: ["localhost:9200"]
5

I dont have elasticsearch set

6

Ah sorry, you are using OOB module. The module is using the ingestion pipeline in ES.

Go to module/o365/audit/manifest.yml, try to comment
#ingest_pipeline: ingest/pipeline.yml

7

Sorry , I dont have the specified folder even

this is 'tree' o/p
sriram@RP-IN-LTP6:/etc/filebeat$ tree
.
├── fields.yml.b
├── filebeat.reference.yml
├── filebeat.yml
└── modules.d
├── :
└── o365.yml

Also, grep command yields nothing

grep -r ingest_pipeline 
sudo grep -r ingest.p
filebeat.reference.yml:  # The ingest pipeline ID associated with this input. If this is set, it
8

/usr/share/filebeat/module/o365/audit/manifest.yml

9

Can you share your entire filebeat.yml formatted with the </> button?

I don't think you need to comment anything in the modules to make it work with the logstash output or file output, you will still get the warning about the ingest pipeline, but as it says, you can ignore it.

Also, when using a module, the parse is done using the ingest pipeline in elasticsearch, if you send it to a file your message will not be parsed or have any enrichment that the ingest pipeline do.

10 
filebeat.inputs:


- input-type: log

  id: my-filestream-id

  enabled: true

  paths:
    - /var/log/*.log




    -  level: debug


filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: true

  reload.period: 10s


setup.template.settings:
  index.number_of_shards: 1










output.file:
  enabled: true

  codec.json:
    pretty: true


  path: "/tmp/filebeat"

  filename: filebeat

  rotate_every_kb: 10000

  number_of_files: 7

  permissions: 0600
  

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


logging.level: debug
11

the /etc/filebeat/modules.d/o365.yml

- module: o365
  audit:
    enabled: true

    var.application_id: "5c36b442-3c39-4101-ab07-zzzzz"

    var.tenants:
     - id: "cxxxxx-3c39xxxxab07-xxxxx"
       name: "zzzz.onmicrosoft.com"

     var.content_type:
      - "Audit.AzureActiveDirectory"
      - "Audit.Exchange"
      - "Audit.SharePoint"
      - "Audit.General"
      - "DLP.All"


    var.client_secret: "xxx~CC4zVqbziaalW2wGBxtJm_xxx8ic3u"
12

BTW that is just a warning has no impact when you use file output.... other than as @leandrojmp mentioned if you are using a modules the data will not be parsed as that usually happens in the elasticsearch with the ingest pipeline

And are you actually trying to run filebeat or are you running filebet setup command

13

I am not concerned about the warning. I wanted filebeat to fetch Azure AD logs for the tenant. I thought the log can give additional information for the folks.
Also, the beat has yet to complain about the authentication (I supplied wrong tenant credentians and still I dont see auth. errors.
Referring to this:

    var.application_id: "5c36b442-3c39-4101-ab07-zzzzz"

    var.tenants:
     - id: "cxxxxx-3c39xxxxab07-xxxxx"
       name: "zzzz.onmicrosoft.com"
14

Ok so what do your filebeat startup logs look like It should show you that it's starting the 0365 module etc and tell you whether it failed to connect...

If you don't see enough there, would you probably can up the filebeat logs to debug level.

Also filebeat console output is a good way to debug... Then you can set up the file afterwards.

15

A few logs:

Sep 12 00:28:02 RP-IN-LTP6 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 1.
Sep 12 00:28:02 RP-IN-LTP6 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Sep 12 00:28:02 RP-IN-LTP6 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
Sep 12 00:28:02 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:02.319+0530","log.origin":{"file.name":"instance/beat.go","file.line":689},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:02 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:02.319+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":747},"message":"Beat metadata path: /var/lib/filebeat/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"warn","@timestamp":"2022-09-12T00:28:05.328+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.328+0530","log.logger":"docker","log.origin":{"file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.328+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":129},"message":"add_cloud_metadata: starting to fetch metadata, timeout=3s","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.353+0530","log.logger":"add_docker_metadata","log.origin":{"file.name":"add_docker_metadata/add_docker_metadata.go","file.line":91},"message":"add_docker_metadata: docker environment detected","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.353+0530","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":212},"message":"Start docker containers scanner","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.353+0530","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":375},"message":"List containers","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.371+0530","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":265},"message":"Fetching events since 2022-09-12 00:28:05.3710785 +0530 IST m=+3.085770229","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.371+0530","log.logger":"add_docker_metadata.bus-docker","log.origin":{"file.name":"bus/bus.go","file.line":88},"message":"map[container:0xc0007220e0 start:true]","service.name":"filebeat","libbeat.bus":"docker","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.371+0530","log.logger":"add_docker_metadata.bus-docker","log.origin":{"file.name":"bus/bus.go","file.line":88},"message":"map[container:0xc000722150 start:true]","service.name":"filebeat","libbeat.bus":"docker","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.371+0530","log.logger":"add_docker_metadata.bus-docker","log.origin":{"file.name":"bus/bus.go","file.line":88},"message":"map[container:0xc000722850 start:true]","service.name":"filebeat","libbeat.bus":"docker","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for openstack after 57.720753ms. result=[provider:openstack, error=failed requesting openstack metadata: Get \"http://169.254.169.254/2009-04-04/meta-data/placement/availability-zone\": dial tcp 169.254.169.254:80: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for aws after 57.8159ms. result=[provider:aws, error=failed requesting aws metadata: Get \"http://169.254.169.254/2014-02-25/dynamic/instance-identity/document\": dial tcp 169.254.169.254:80: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for huawei after 57.837314ms. result=[provider:huawei, error=failed requesting huawei metadata: Get \"http://169.254.169.254/openstack/latest/meta_data.json\": dial tcp 169.254.169.254:80: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for gcp after 57.852363ms. result=[provider:gcp, error=failed requesting gcp metadata: Get \"http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json\": dial tcp 169.254.169.254:80: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for openstack after 57.877865ms. result=[provider:openstack, error=failed requesting openstack metadata: Get \"https://169.254.169.254/2009-04-04/meta-data/instance-id\": dial tcp 169.254.169.254:443: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for azure after 57.891088ms. result=[provider:azure, error=failed requesting azure metadata: Get \"http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02\": dial tcp 169.254.169.254:80: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":166},"message":"add_cloud_metadata: received disposition for digitalocean after 57.904246ms. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get \"http://169.254.169.254/metadata/v1.json\": dial tcp 169.254.169.254:80: connect: no route to host, metadata={}]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/providers.go","file.line":132},"message":"add_cloud_metadata: fetchMetadata ran for 57.923648ms","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":120},"message":"Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map[], add_cloud_metadata={}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], add_kubernetes_metadata","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.386+0530","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":117},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","clone3","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rseq","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}},"ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.387+0530","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.387+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1067},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"8ec7b8cc-1ecd-4dab-bcb7-4e207ef8d455"},"ecs.version":"1.6.0"}}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.387+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"7826dc5e91c6e6d2487e05d3a8298f49041cd5c2","libbeat":"8.2.3","time":"2022-06-08T15:51:35.000Z","version":"8.2.3"},"ecs.version":"1.6.0"}}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.387+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1079},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.18.2"},"ecs.version":"1.6.0"}}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":91},"message":"registering decoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":94},"message":"registering decoder 'application/x-ndjson': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":97},"message":"registering decoder 'text/csv': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.input-type filebeat.inputs.0.paths.0 filebeat.inputs.0.paths.1.level]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"warn","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":89},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 0 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":167},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.397+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.401+0530","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":193},"message":"Uptime: 3.109237752s","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.401+0530","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":160},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.401+0530","log.logger":"add_docker_metadata","log.origin":{"file.name":"docker/watcher.go","file.line":313},"message":"Watcher stopped","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.402+0530","log.origin":{"file.name":"instance/beat.go","file.line":467},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: {"log.level":"error","@timestamp":"2022-09-12T00:28:05.402+0530","log.origin":{"file.name":"instance/beat.go","file.line":1042},"message":"Exiting: Failed to start crawler: starting input failed: error while initializing input: can not convert 'object' into 'string' accessing 'filebeat.inputs.0.paths.1' (source:'/etc/filebeat/filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228627]: Exiting: Failed to start crawler: starting input failed: error while initializing input: can not convert 'object' into 'string' accessing 'filebeat.inputs.0.paths.1' (source:'/etc/filebeat/filebeat.yml')
Sep 12 00:28:05 RP-IN-LTP6 systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
Sep 12 00:28:05 RP-IN-LTP6 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Sep 12 00:28:05 RP-IN-LTP6 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 2.
Sep 12 00:28:05 RP-IN-LTP6 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Sep 12 00:28:05 RP-IN-LTP6 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228781]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.812+0530","log.origin":{"file.name":"instance/beat.go","file.line":689},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228781]: {"log.level":"debug","@timestamp":"2022-09-12T00:28:05.812+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":747},"message":"Beat metadata path: /var/lib/filebeat/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
Sep 12 00:28:05 RP-IN-LTP6 filebeat[1228781]: {"log.level":"info","@timestamp":"2022-09-12T00:28:05.812+0530","log.origin":{"file.name":"instance/beat.go","file.line":697},"message":"Beat ID: 8ec7b8cc-1ecd-4dab-bcb7-4e207ef8d455","service.name":"filebeat","ecs.version":"1.6.0"}
16

Well it looks you never got to the O365 module because you have an syntax error in your filebeat.yml

Pretty straight forward

- input-type: log

  id: my-filestream-id

  enabled: true

  paths:
    - /var/log/*.log




    -  level: debug <!---- This does not belong here! Not sure what you are trying but that is not proper

I think perhaps you were trying to set the logging level?

The filebeat logs are pretty decent... if you look at them they will direct you to the issue.

Also why you you have that input enabled if you are trying to get o365 logs?

1 Like
17

thanks @stephenb ! I got this resolved by commenting the filebeat.inputs section

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
#- input-type: log

  # Unique ID among all inputs, an ID is required.
  #  id: my-filestream-id

  # Change to true to enable this input configuration.
  #enabled: true

  # Paths that should be crawled and fetched.

The main gripe still is, the configuration model is not intuitive ,from an outsider's (not part of beat dev) perspective. Call me dumb :slight_smile:

18

Nope not just you.. takes a bit to get used to ...

Basically there are

Inputs, outputs and processor

If you use a module
It is an input + template (schema) + ingest pipeline (parsing) + dashboards.

Oh then of course you add got .yml into it .... All bets are off :slight_smile:

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.