Filebeat nginx module does not work with AWS ES

Elastic Stack Beats
1

Filebeat nginx module does not work with AWS ES due to lack of support for ingest-geoip plugin in AWS ES.

Is there any temporary workaround for this issue ? I don't need the geoip information for now.

Also any update on https://github.com/elastic/beats/issues/10867 ?

Thanks
John

2

Hi @john_eapen, the github issue still under discussion. We will update it with conclusion.
Meantime, maybe you can disable geoip manually by removing the processor in https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json#L126 and rebuild filebeat binary.

3

Hi Kaiyan,

Thanks for you reply.
But this did not work for me. Can you tell me if the following is supposed to work or am I missing something?

We use docker/kubernetes. Instead of rebuilding the entire binary, i adjusted the Dockerfile as follows to include the modified "nginx/access/ingest/default.json" where i removed all references to geoip processor.

cat docker/Dockerfile
FROM docker.elastic.co/beats/filebeat-oss:7.4.1
COPY fields.yml /usr/share/filebeat/fields.yml
COPY module/nginx/access/ingest/default.json
/usr/share/filebeat/module/nginx/access/ingest/default.json #Added this line
USER root

With this docker image, I checked the running pod and the ingest file does not the geoip references.
But it did not work.

Thanks

4

Hmmm did you build filebeat-oss from the source code?
Maybe something like https://github.com/elastic/beats/blob/master/filebeat/tests/open-file-handlers/Dockerfile-filebeat

5

Hi Kaiyan,

Appreciate your help. Sorry could not reply to you earlier.

did you build filebeat-oss from the source code?
Not I did not build the entire filebeat-oss from source code. But as I mentioned earlier, I made sure that the "module/nginx/access/ingest/default.json" that is used by filebeat does not have any references to "geo-ip". I did that by including this modified default.json into my own dockerfile/build.

My question is,
Is it necessary to rebuilt filebeat-oss ? Is the nginx ingest/default.json not read from the file by filebeat at startup. ?

Thanks
John

6

Sorry I just came back from vacation. I think the way you are doing will work. But there are more than nginx module is using geoip and I believe all the modules that uses geoip needs to have a new/updated default.json file. Maybe give that a try?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.