# Packetbeat Logstash and Elastic cloud

**URL:** <https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034>\
**Category:** Beats\
**Tags:** packetbeat\
**Created:** [February 23, 2022, 12:32pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034 "2022-02-23T12:32:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kasuma](https://avatars.discourse-cdn.com/v4/letter/k/3e96dc/32.png) [@kasuma](https://discuss.elastic.co/u/kasuma)\
**Post date:** [February 23, 2022, 12:32pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034/1 "2022-02-23T12:32:08Z")

</div>

Hello,

I have a problem were my dashboard showing an "error" message when data are loaded from logstash, but if i load data directly from packetbeat to elastic cloud everything is working

This is my setup:

packetbeat.yml:

> packetbeat.interfaces.device: any  
> packetbeat.interfaces.buffer\_size\_mb: 100  
> packetbeat.interfaces.internal\_networks:
> 
> - private  
> packetbeat.flows:  
> timeout: 30s  
> period: 10s  
> packetbeat.protocols:
> 
> - type: icmp
> 
> - type: dhcpv4  
> ports: [67, 68]
> 
> - type: dns  
> ports: [53]
> 
> - type: http  
> ports: [80, 8080]
> 
> - type: tls  
> ports:

logstash.conf:

> input {  
> beats {  
> port =\> 5044  
> ssl =\> true  
> ecs\_compatibility =\> "v8"  
> ssl\_certificate =\> "/etc/ssl/logstash/ca.crt"  
> ssl\_key =\> "/etc/ssl/logstash/x.key"  
> ssl\_verify\_mode =\> "none"  
> }  
> }
> 
> filter {  
> }
> 
> output {  
> elasticsearch {  
> cloud\_id =\> "x"  
> cloud\_auth =\> "x"  
> index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
> template =\> "/usr/share/logstash/config/packetbeat.template.json"  
> template\_name =\> "packetbeat-8.0.0"  
> template\_overwrite =\> true  
> }  
> }  
> To get the json template, i exported it from packetbeat with export command

I also run the command:

> packetbeat -e setup --index-management -E output.logstash.enabled=false -E cloud.id="x" -E cloud.auth="x"

And add the dashboards:

> packetbeat -e setup --dashboards -E output.logstash.enabled=false -E cloud.id="x" -E cloud.auth="x"

Now, if i go to kibana, my Indices are showing up:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5dd26f3c0538ed7884220f82161e7c4d99baeeb4.png)

My index templates is set:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1d33ff31e5bf895c3da14f612fad6eecda5ba9ed.png)

Data are showing up under "packetbeat-\*"

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3de2c7f9212c5543e08cd9e4f007fcf5bf406567.png)

Data are OK:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b21442cb7d89d33a39ba6c35472544b45c6df2ba.png)

But if i go to the dashboards, i see:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/3/33f538e3967528d10800957c87f8bd91e80260ec.png)

If i check the Index patterns, i only have 293 fields:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/705fa5d09dc60334fb380985e3242e1371cdbc20.png)

Now, if i try to change the output on packetbeat.yml to Elactic cloud instead of logstash and delete the old indices generated by logstash, all my dashboard are showing up and working, i also see more than 1000 fields inside the index pattern 'packetbeat-\*'

I don't understand why i see this error as i think i followed correctly the how-to.

Thanks you very much if you can help

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 23, 2022, 2:48pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034/2 "2022-02-23T14:48:15Z")

</div>

Perhaps take a look at this...

> [@Help/Advice needed setting up geo-ip filters in an on-prem Logstash to SIEM in Elastic Cloud instance](https://discuss.elastic.co/t/help-advice-needed-setting-up-geo-ip-filters-in-an-on-prem-logstash-to-siem-in-elastic-cloud-instance/273593/7):
>
> Hi @ronmer Thanks for the detailed explanation and questions. Quick Recap: So you want architecture B cool that is very common. Packetbeat -\> Logstash -\> Elasticsearch in Elastic Cloud With this architecture you do not need to do the GEO IP in Logstash if you are using Packetbeat. (I will show you below) With Logstash on prem you can have the exact same experience as you had on cloud following simple instructions below, very very little work. Good you understand Private IP. Y…

---

<div class="post-metadata">

**Author:** ![kasuma](https://avatars.discourse-cdn.com/v4/letter/k/3e96dc/32.png) [@kasuma](https://discuss.elastic.co/u/kasuma)\
**Post date:** [February 23, 2022, 3:31pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034/3 "2022-02-23T15:31:56Z")

</div>

Thanks you very much @stephenb ! this seem much clear now, you saved my day 🙂

with this configuration i'm still seeing error on logstash such as:

```auto
 Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"packetbeat-8.0.0"
...
 "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams

```

I'm using packetbeat 8.0 and logstash 8.0, but my ELK cloud instance is on 7.17

Maybe you will have a suggestion about this?

Thanks !

---

<div class="post-metadata">

**Author:** ![kasuma](https://avatars.discourse-cdn.com/v4/letter/k/3e96dc/32.png) [@kasuma](https://discuss.elastic.co/u/kasuma)\
**Post date:** [February 23, 2022, 3:41pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034/4 "2022-02-23T15:41:47Z")

</div>

It worked by adding:  
`action => "create"`  
here:

```auto
 else {
    elasticsearch {
      cloud_auth => "x"
      cloud_id => "x"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 23, 2022, 3:42pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034/5 "2022-02-23T15:42:03Z")

</div>

Yeah the Beats and Logstash should really not be a Major Version Ahead... not sure that is the problem but it could be.

Oh @kasuma Welcome to the community!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2022, 5:42pm UTC](https://discuss.elastic.co/t/packetbeat-logstash-and-elastic-cloud/298034/6 "2022-03-23T17:42:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.