Custom beats not writing logs in file when starting as service

(Debasish283) #1

Hi All,

We have build custom beats for our use case and able to start the service successfully but logs are not writing to the specified log path (var/log/beats)

Below is the config of beat.yml,

beat:
  # Defines how often an event is sent to the output
    period: 300s
    zohoAuthToken: 'aadwsjhwkwkb1cb'
    monitorID:
      - '159205000000090003'
      - '159205000010259001'
    granularity: '1'
    periodAPI: '0'

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: "beat"

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:31055"]

#================================ Logging =====================================
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/site24x7beat
  name: site24x7beat
  keepfiles: 7
  permissions: 0644

If I start the beat as like below and its writing but when I start it as service logs are not writing to the file,

sudo beat -c /etc/beat/beat.yml -v -d "*"

Below is the logs info,

2019-05-14T08:14:42.799Z        INFO    [beat]  instance/beat.go:893    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.11.5"}}}
2019-05-14T08:14:42.800Z        INFO    [beat]  instance/beat.go:897    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-05-14T03:51:56Z","containerized":false,"name":"SmartThink-Demo","ip":["127.0.0.1/8","::1/128","10.0.2.5/24","fe80::20d:3aff:fea0:9e96/64","172.17.0.1/16","10.244.0.0/32","fe80::38c7:2cff:feda:1940/64","10.244.0.1/24","fe80::28fb:faff:fefe:d354/64","fe80::a8ea:4ff:fe37:c5c8/64","fe80::60fd:a3ff:fe6c:41d8/64","fe80::7470:cdff:fe08:23fc/64","fe80::289b:3ff:feb6:4487/64","fe80::a4a4:27ff:fe78:3ca5/64","fe80::1033:fdff:fe05:4c2a/64"],"kernel_version":"4.15.0-1041-azure","mac":["00:0d:3a:a0:9e:96","02:42:b0:6b:ab:83","3a:c7:2c:da:19:40","0a:58:0a:f4:00:01","aa:ea:04:37:c5:c8","62:fd:a3:6c:41:d8","76:70:cd:08:23:fc","2a:9b:03:b6:44:87","a6:a4:27:78:3c:a5","12:33:fd:05:4c:2a"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.5 LTS (Xenial Xerus)","major":16,"minor":4,"patch":5,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0,"id":"fc6d4988a3184dc4b78815ada9b48a46"}}}
2019-05-14T08:14:42.801Z        INFO    [beat]  instance/beat.go:926    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/usr/share/beat/bin", "exe": "/usr/share/beat/bin/beat", "name": "beat", "pid": 20972, "ppid": 20970, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-05-14T08:14:42.180Z"}}}
2019-05-14T08:14:42.801Z        INFO    instance/beat.go:283    Setup Beat: beat; Version: 8.0.0
2019-05-14T08:14:42.801Z        DEBUG   [beat]  instance/beat.go:309    Initializing output plugins
2019-05-14T08:14:42.801Z        DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2019-05-14T08:14:42.801Z        INFO    [publisher]     pipeline/module.go:97   Beat name: beat
2019-05-14T08:14:42.801Z        INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2019-05-14T08:14:42.801Z        INFO    instance/beat.go:405    beat start running.
2019-05-14T08:14:42.801Z        INFO    beater/beat.go:246      beat is running! Hit CTRL-C to stop it.

Please let me know your thoughts.

Thanks,
Debasish

(Andrew Kroh) #2

If you’re using systemd then the logs are probably going to journald in order to act like the other services on the host. But you can modify the service’s unit file to remove the -e flag. See https://www.elastic.co/guide/en/beats/filebeat/master/running-with-systemd.html.