How can I add the request IP address to every document?

Elastic Stack Elasticsearch
1

Is it possible to add the IP addressed used (in the HTTP request) to the document? I want to do this to then use the geoip processor in a pipeline to add location information?

2

Hi,

Your request is not really clear? Are you using Beats, LogStash or another application to gather the information?

Beats has a prcoessor to add host information: https://www.elastic.co/guide/en/beats/filebeat/7.8/add-host-metadata.html

Best regards
Wolfram

3

Hi Wolfram,

I am adding the data to Elasticsearch using the REST API. I would like to get the IP address from the HTTP request used to make the REST API request.

4

Hi,

I am not sure this is possible: In theory, you would have to create an ingest pipeline in ElasticSearch which does what you want and modify your REST call to execute the pipeline. But from what I found there is no processor for adding the IP from the REST call. See here for the Github issue.
Depending on the usecase this might also be problematic as the IP adress can be faked.

I guess you have (at least) 3 options depending on your infrastructure:
Option 1: Alter your REST call to include the IP adress out of the box
Option 2: Switch from a direct REST call to ElasticSearch to LogStash. LogStash supports this.
Option 3: This is not exactly what you asked for as it does not directly give you the IP. You may use the set-security-user-processor to get the informationabout the user/apiKey which was used to write send the document. If each server has a separate user/apiKey you can use this information to uniquely identify the sending server(You may then use a mapping to get the IP based on the server but it is not the same as getting the IP of the server obviously)

Best regards
Wolfram

5

Dear Wolfram,

Thanks for your reply. I suppose I will have to add the IP address to the REST call, as using REST is required - I don't think I can use LogStash.

6

Hi Wolfram,

Should I be using Logstash or REST API if adding data from (possibly millions) of client devices (think IoT).

At the moment I am using the REST API however I think potentially I should have used logstash instead?

Is there anywhere that compares the two methods and the advantages/disadvantages?

7

Hi,

In my opinion - if you do not yet have LogStash installed and you are able to modify the sender to provide the IP adress do that. Otherwise you would get more complexity by managing a new Software(LogStash), its Pipelines, etc...

I could not find much (it either was not in english or relatively old) but this might give you at least an overview: Elasticsearch data collection and processing - Logstash VS Ingest Node - Programmer Sought

Best regards
Wolfram

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.