GeoIP enrichment not working

Elastic Stack Beats
21

The Cloud Auth / Id looks good

Nope

output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true
  pipeline: geoip-info

or shorthand

output.elasticsearch.enabled: true
output.elasticsearch.pipeline: geoip-info

gotta run ... good luck!

22

uugghhhh... it is killing me ... :smiley:

This won't let me sleep.

Have a good one!

23

Check this out :smiley:

image
image857×313 81.7 KB

1 Like
24

NICE!!! What was the fix?

If you want to run it through logstash we can do that but I am OOO rest of the day.

I can show you how... it is in those links above ...

25

There was none... I knew this was not conf issue, because it worked with the same configuration on your end. It was not, firewall issue, because beat was able to connect and send traffic to elastic.

I got frustrated and I set up a new VM with win 10. I did exactly same thing and it worked as a charm. This makes me think that maybe the service that was initially set up was not fully uninstalled after reconfiguration ... who knows, I am glad it works.

Sure, if you have time I will be happy to learn how to do this tomorrow. I will read links you provided, and do my best to set this up. If I bump into problems, I will reach out to you.

26

To use Logstash in the packetbeat.yml.

  1. comment out the elasticsearch output and cloud auth id

  2. set the logstash output and pipeline

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
  pipeline : geoip-info
  1. Use this for your logstash.conf.
    This acts as a pass through and forwards the pipeline definition

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      cloud_auth => "elastic:password"
      cloud_id => "mycloud:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRj......"

      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
    }
  } else {
    elasticsearch {
      cloud_auth => "elastic:password"
      cloud_id => "mycloud:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRj......"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    }
  }
}
1 Like
27

Yes, this is exactly how I have logstash.conf set up.

Here I have a question. My understanding is that if elastic has 2,3,5...20 pipelines, it needs to separate them. Does it mean that it also needs to balance traffic? How does it choose which pipeline to take care of first? Lets say I have 4 different clusters on different pipelines to make it more interesting one sending traffic from Canada one from the USA, and they both send traffic to the same Elastic cloud.

28

I assume you mean My understanding is that if Logstash has 2,3,5...20 pipelines, i....

In short please open seperate topic on these questions there are some very good Logstash people on the forum.

I would also suggest some of these topics in the documentation.

If you you are really saying how ingest pipelines work in elasticsearch in Elastic Cloud it is a distributed system, with a proxy, with a multiple queues, first come first serve, with prioritization and parallelization.... and a whole lot of configuration settings. If you want more than that you are probably going to need to look at the code, or perform some actual setup and then ask specific questions.

And course proper configuration like using coordinators scaling and performance tuning are part of this equation as well

Just for clarity if you run a self managed cluster, best practice is to load balance request, but even with that understanding how elastic works under the cover with distributing index and query might be worth learning a bit. This is super old but still very good for those concepts but be careful syntax is sometimes no longer valid.

Also in your questions since there are logstash pipelines and ingest pipeline please be specific to which you are referring to as both can be used or not

Good Luck come back with some specifics.

1 Like
29

I have a question over here. Logstash .yml has an entry under Pipeline settings with pipeline.id: main
Having this in mind, we want to push logs from packet beat through logstash using geoip-info pipeline. At the same time I am using different beats to send logs using logstash with main pipeline. How do I configure logstash to handle this ? Can I simply do pipeline.id: main, geoip-info ?

30

You are confusing and ingest pipeline with logstash pipeline... 2 different things completely, but often confused.

The pipelines referred to in logstasg pipelines.yml are logstash pipelines not the geoip-info pipelines.

Highly encourage you to open specific threads on specific logstash questions.

Logstash has pipelines...

In those pipelines in the output section you can define to run an ingest pipeline in elasticsearch

This code above

      pipeline => "%{[@metadata][pipeline]}"

Says run the ingest pipeline I defined in the beats in elastic.
That whole logstash.conf is a logstash pipeline.

31

Got you that makes perfect sense. Also, I moved logstash problem to another threat.
https://discuss.elastic.co/t/logstash-communication-configuration/275331

32

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.