Auditbeat index pattern

I'm not able to create the index pattern and tried the following steps.
what am i doing wrong?

Tried 1:

auditbeat setup --template -E output.logstash.enabled=false -E output.elasticsearch.hosts= -E setup.kibana.username: -E setup.kibana.password:

Output:

Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://xxxxx:9243: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}}],"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}},"status":401}]

Tried 2:

Config:

    setup.kibana:
   dashboards.index: "auditbeat-*"
   username: 
   password: 
    dashboards.enabled: true
13/06/2019 10:18:34Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://xxxx.xxxxx:9243: Failed to parse JSON response: invalid character '<' looking for beginning of value]

Hi @yasin - have you configured username and password for both Kibana and Elasticsearch? In the config you pasted it looks like only Kibana. Do you mind pasting the output of ./auditbeat export config (with any sensitive data stripped out)?

Dear Cwurm,

Would like to thank you for your swift reply.
Below as requested:

auditbeat:
  config:
    modules:
      path: /usr/share/auditbeat/conf.d/*.yml
      reload:
        enabled: true
        period: 10s
  max_start_delay: 10s
  modules:
  - audit_rule_files:
    - /usr/share/auditbeat/audit.rules.d/*.conf
    audit_rules: |
      ## Define audit rules here.
      ## Create file watches (-w) or syscall audits (-a or -A). Uncomment these
      ## examples or add your own rules.

      ## If you are on a 64 bit platform, everything should be running
      ## in 64 bit mode. This rule will detect any use of the 32 bit syscalls
      ## because this might be a sign of someone exploiting a hole in the 32
      ## bit API.
      #-a always,exit -F arch=b32 -S all -F key=32bit-abi

      ## Executions.
      -a always,exit -F arch=b64 -S execve,execveat -k exec

      ## External access (warning: these can be expensive to audit).
      -a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access

      ## Identity changes.
      -w /etc/group -p wa -k identity
      -w /etc/passwd -p wa -k identity
      -w /etc/gshadow -p wa -k identity

      ## Unauthorized access attempts.
      -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
      -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
    backlog_limit: 8196
    failure_mode: silent
    include_raw_message: false
    include_warnings: false
    module: auditd
    rate_limit: 0
    resolve_ids: true
  - module: file_integrity
    paths:
    - /bin
    - /usr/bin
    - /sbin
    - /usr/sbin
    - /etc
dashboards:
  beat: auditbeat
  enabled: true
host: 
index: auditbeat-*
output:
  elasticsearch:
    hosts: 
password: 
path:
  config: /usr/share/auditbeat
  data: /usr/share/auditbeat/data
  home: /usr/share/auditbeat
  logs: /usr/share/auditbeat/logs
setup:
  kibana: null
username:

Dear Cwurm,

Now i'm getting the following message with the same config i've send you above:

13/06/2019 11:51:012019-06-13T09:51:01.806Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://fa4e0d8a298c4b70b1b5f68fb60109df.eu-west-1.aws.found.io:9243)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

13/06/2019 11:51:012019-06-13T09:51:01.807Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://fa4e0d8a298c4b70b1b5f68fb60109df.eu-west-1.aws.found.io:9243)) with 29 reconnect attempt(s)

13/06/2019 11:51:012019-06-13T09:51:01.807Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer

13/06/2019 11:51:012019-06-13T09:51:01.807Z INFO [publish] pipeline/retry.go:191 done

13/06/2019 11:51:012019-06-13T09:51:01.807Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer

13/06/2019 11:51:012019-06-13T09:51:01.807Z INFO [publish] pipeline/retry.go:168 done

13/06/2019 11:51:282019-06-13T09:51:28.630Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"auditd":{"received_msgs":25},"beat":{"cpu":{"system":{"ticks":1760},"total":{"ticks":4940,"time":{"ms":12},"value":4940},"user":{"ticks":3180,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":15},"info":{"ephemeral_id":"5609164f-b284-4898-bb1e-df6892817d1b","uptime":{"ms":1140072}},"memstats":{"gc_next":55759488,"memory_alloc":28258832,"memory_total":194397256}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"read":{"bytes":704},"write":{"bytes":167}},"pipeline":{"clients":2,"events":{"active":4117,"retry":50}}},"system":{"load":{"1":0.22,"15":0.12,"5":0.17,"norm":{"1":0.055,"15":0.03,"5":0.0425}}}}}}

It looks like your config might not have user and password set up correctly. The elasticsearch output section should look something like this (note the username and password nested underneath elasticsearch):

output:
  elasticsearch:
    hosts:
    - localhost:9200
    password: changeme
    username: elastic

How did you set up your config? The easiest is usually to take the auditbeat.yml file Auditbeat ships with and uncomment any sections you need.

Dear Cwurm,

I'm a bit further but now i got this: --> below log i've added my config

3/06/2019 13:06:352019-06-13T11:06:35.807Z WARN elasticsearch/client.go:539 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x38074d62, ext:63696020794, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"event":common.MapStr{"module":"file_integrity", "dataset":"file", "action":[]string{"created"}}, "file":common.MapStr{"owner":"root", "ctime":common.Time{wall:0x2468397a, ext:63690061616, loc:(*time.Location)(nil)}, "uid":0x0, "gid":0x0, "mode":"0755", "group":"root", "mtime":common.Time{wall:0x0, ext:63684381344, loc:(*time.Location)(nil)}, "size":0xe598, "type":"file", "inode":"551793", "path":"/usr/sbin/zic"}, "hash":common.MapStr{"sha1":"5e90ea8d94f86de2dfbea9ea710117f7c1a6c1cf"}, "beat":common.MapStr{"name":"f13a9e01f8ad", "hostname":"f13a9e01f8ad", "version":"6.7.2"}, "host":common.MapStr{"name":"f13a9e01f8ad"}}, Private:interface {}(nil)}, Flags:0x0} (status=403): {"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [yasin.donmez]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don't support replacing their indices"}}

13/06/2019 13:06:352019-06-13T11:06:35.807Z WARN elasticsearch/client.go:539 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x380e8c72, ext:63696020794, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"event":common.MapStr{"dataset":"file", "action":[]string{"created"}, "module":"file_integrity"}, "file":common.MapStr{"type":"file", "gid":0x0, "inode":"551794", "mtime":common.Time{wall:0x0, ext:63676536532, loc:(*time.Location)(nil)}, "ctime":common.Time{wall:0x2468397a, ext:63690061616, loc:(*time.Location)(nil)}, "uid":0x0, "group":"root", "mode":"0755", "owner":"root", "path":"/usr/sbin/zramctl", "size":0x14928}, "hash":common.MapStr{"sha1":"0c2ea7fbd12728a96fe15be9ae2c8bf80c24b263"}, "beat":common.MapStr{"name":"f13a9e01f8ad", "hostname":"f13a9e01f8ad", "version":"6.7.2"}, "host":common.MapStr{"name":"f13a9e01f8ad"}}, Private:interface {}(nil)}, Flags:0x0} (status=403): {"type":"security_exception","reason":"action [indices:admin/create] is unauthorized for user [yasin.donmez]","caused_by":{"type":"illegal_state_exception","reason":"There are no external requests known to support wildcards that don't support replacing their indices"}}

13/06/2019 13:07:002019-06-13T11:07:00.517Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":390,"time":{"ms":396}},"total":{"ticks":960,"time":{"ms":968},"value":0},"user":{"ticks":570,"time":{"ms":572}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"179d5a69-d24b-4e2c-a983-c6004847cbe4","uptime":{"ms":30022}},"memstats":{"gc_next":9764464,"memory_alloc":6716336,"memory_total":49626416,"rss":32071680}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"batches":15,"dropped":711,"total":711},"read":{"bytes":14094},"type":"elasticsearch","write":{"bytes":382348}},"pipeline":{"clients":2,"events":{"active":0,"published":711,"retry":50,"total":711},"queue":{"acked":711}}},"metricbeat":{"file_integrity":{"file":{"events":711,"success":711}}},"system":{"cpu":{"cores":4},"load":{"1":0,"15":0.07,"5":0.05,"norm":{"1":0,"15":0.0175,"5":0.0125}}}}}}
auditbeat:
  config:
    modules:
      path: /usr/share/auditbeat/conf.d/*.yml
      reload:
        enabled: true
        period: 10s
  max_start_delay: 10s
  modules:
  - audit_rule_files:
    - /usr/share/auditbeat/audit.rules.d/*.conf
    audit_rules: |
      ## Define audit rules here.
      ## Create file watches (-w) or syscall audits (-a or -A). Uncomment these
      ## examples or add your own rules.

      ## If you are on a 64 bit platform, everything should be running
      ## in 64 bit mode. This rule will detect any use of the 32 bit syscalls
      ## because this might be a sign of someone exploiting a hole in the 32
      ## bit API.
      #-a always,exit -F arch=b32 -S all -F key=32bit-abi

      ## Executions.
      #-a always,exit -F arch=b64 -S execve,execveat -k exec

      ## External access (warning: these can be expensive to audit).
      #-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access

      ## Identity changes.
      #-w /etc/group -p wa -k identity
      #-w /etc/passwd -p wa -k identity
      #-w /etc/gshadow -p wa -k identity

      ## Unauthorized access attempts.
      #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
      #-a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
    backlog_limit: 8196
    failure_mode: silent
    include_raw_message: false
    include_warnings: false
    module: auditd
    rate_limit: 0
    resolve_ids: true
  - exclude_files:
    - (?i)\.sw[nop]$
    - ~$
    - /\.git($|/)
    hash_types:
    - sha1
    max_file_size: 100 MiB
    module: file_integrity
    paths:
    - /bin
    - /usr/bin
    - /sbin
    - /usr/sbin
    - /etc
    recursive: false
    scan_at_start: true
    scan_rate_per_sec: 50 MiB
logging:
  files: null
  to_files: true
output:
  elasticsearch:
    enabled: true
    hosts: https://:9243
    index: auditbeat-*
    password: 
    protocol: https
    username: 
path:
  config: /usr/share/auditbeat
  data: /usr/share/auditbeat/data
  home: /usr/share/auditbeat
  logs: /usr/share/auditbeat/logs
setup:
  kibana:
    host: https://:9243
    password: 
    protocol: https
    username: 
  template:
    enabled: true
    name: auditbeat-%{[beat.version]}
    pattern: auditbeat-%{[beat.version]}-*
    settings: null

Dear Cwurm,

By adding the correct Index it worked:

auditbeat-%%{[beat.version]}-%{+yyyy.MM.dd} instead of auditbeat-%%{[agent.version]}-%{+yyyy.MM.dd}

Great to hear, glad you got it working!

1 Like