Packetbeat Logstash and Elastic cloud

[kasuma (Olivier)]

Hello,

I have a problem were my dashboard showing an "error" message when data are loaded from logstash, but if i load data directly from packetbeat to elastic cloud everything is working

This is my setup:

packetbeat.yml:

packetbeat.interfaces.device: any
packetbeat.interfaces.buffer_size_mb: 100
packetbeat.interfaces.internal_networks:

• private
  packetbeat.flows:
  timeout: 30s
  period: 10s
  packetbeat.protocols:

• type: icmp

  enabled: true

• type: dhcpv4
  ports: [67, 68]

• type: dns
  ports: [53]

• type: http
  ports: [80, 8080]

• type: tls
  ports:

  • 443
    output.logstash:
    hosts: ["x.io:5044"]
    ssl.certificate_authorities: ["/etc/packetbeat/ca.crt"]
    processors:
  • if.contains.tags: forwarded
    then:
    • drop_fields:
      fields: [host]
      else:
    • add_host_metadata: ~
  • add_cloud_metadata: ~
  • add_docker_metadata: ~
  • detect_mime_type:
    field: http.request.body.content
    target: http.request.mime_type
  • detect_mime_type:
    field: http.response.body.content
    target: http.response.mime_type
    setup.kibana:

logstash.conf:

input {
beats {
port => 5044
ssl => true
ecs_compatibility => "v8"
ssl_certificate => "/etc/ssl/logstash/ca.crt"
ssl_key => "/etc/ssl/logstash/x.key"
ssl_verify_mode => "none"
}
}

filter {
}

output {
elasticsearch {
cloud_id => "x"
cloud_auth => "x"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
template => "/usr/share/logstash/config/packetbeat.template.json"
template_name => "packetbeat-8.0.0"
template_overwrite => true
}
}
To get the json template, i exported it from packetbeat with export command

I also run the command:

packetbeat -e setup --index-management -E output.logstash.enabled=false -E cloud.id="x" -E cloud.auth="x"

And add the dashboards:

packetbeat -e setup --dashboards -E output.logstash.enabled=false -E cloud.id="x" -E cloud.auth="x"

Now, if i go to kibana, my Indices are showing up:

My index templates is set:

Data are showing up under "packetbeat-*"

Data are OK:

But if i go to the dashboards, i see:

If i check the Index patterns, i only have 293 fields:

Now, if i try to change the output on packetbeat.yml to Elactic cloud instead of logstash and delete the old indices generated by logstash, all my dashboard are showing up and working, i also see more than 1000 fields inside the index pattern 'packetbeat-*'

I don't understand why i see this error as i think i followed correctly the how-to.

Thanks you very much if you can help

[stephenb (Stephen Brown)]

Perhaps take a look at this...

Help/Advice needed setting up geo-ip filters in an on-prem Logstash to SIEM in Elastic Cloud instance

Hi @ronmer Thanks for the detailed explanation and questions. Quick Recap: So you want architecture B cool that is very common. Packetbeat -> Logstash -> Elasticsearch in Elastic Cloud With this architecture you do not need to do the GEO IP in Logstash if you are using Packetbeat. (I will show you below) With Logstash on prem you can have the exact same experience as you had on cloud following simple instructions below, very very little work. Good you understand Private IP. Y…

[kasuma (Olivier)]

Thanks you very much @stephenb ! this seem much clear now, you saved my day

with this configuration i'm still seeing error on logstash such as:

 Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"packetbeat-8.0.0"
...
 "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams

I'm using packetbeat 8.0 and logstash 8.0, but my ELK cloud instance is on 7.17

Maybe you will have a suggestion about this?

Thanks !

[kasuma (Olivier)]

It worked by adding:
action => "create"
here:

 else {
    elasticsearch {
      cloud_auth => "x"
      cloud_id => "x"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"

[stephenb (Stephen Brown)]

Yeah the Beats and Logstash should really not be a Major Version Ahead... not sure that is the problem but it could be.

Oh @kasuma Welcome to the community!