Hi there. I am trying to create a region map so that I can see the activity of users from different countries. I have an ip address field in my index and what I am trying to do is map it to geoip location to get fields like location, latitude , longitude and etc. I have a source feeding the data directly to elasticsearch from filebeat and I am running everything on elastic cloud. How do I map my ip address internally, I mean without touching filebeat or any files but within Kibana or elasticsearch so that I will get this field in my index? Thank you.
Sounds like you need to send your data to Logstash first to do the geoip lookups and then ingest it into Elasticsearch. Geo-IP data is pulled from a database, which is not something Elasticsearch does as far as I know.
Elasticsearch also features a geoip processor nowadays, there is no need to use logstash for that, unless you need to do further processing.
1 Like
Ok, great, thanks! Just to clarify on that article you gave link to: it will only give instance of 1 ip address if following the way described there. However, what I need is something similar to automated procession of these ips. Am I wrong or misunderstanding something?
I'm sorry, I am not sure I get the question. Can you elaborate what you mean with automated procession? What exactly are you trying to do?
I am trying to map the field that contains ip addresses , although they are of the type "string", to geoip so that I will have a field (or fields) representing the geo location of these ip addresses in order to build map visualization.
yes, the geoip processor will create additional fields in the JSON document, that need to be mapped properly. You should use the simulate pipeline API to figure out the structure of your resulting JSON document and then map the fields properly in your index before doing any indexation.
How would I map ip field in my index to get new fields?
can you be more specific where your problem is? Take a look at index templates and the mapping datatypes, then try to create a proper template, and let's iterate from there if it is not working.
Yes , sorry , I will be more specific. So, I have managed to do the simulation of pipelines and it works correctly, I get the geoip output with country, latitude, longitude and etc. Now, what I need is to actually create these geoip simulated fields in my already existing index. How should I approach this?
you can use the Put Mapping API to update this in your index, if the fields have not been added yet.
How they would have been added if I did the pipeline simulation API? It is just an example of what I will get. Can you be more exact , in terms of updating the index, for example, following all the documentation that is what I get for simulated API:
1)
PUT _ingest/pipeline/~testing name~
{
"processors":[
{
"grok":
{
"field": "message",
"patterns": ["~some pattern~"]
}
},
{
"geoip":{
"field": "~name of appropriate field~"
}
}
]
}
POST _ingest/pipeline/~testing name~/_simulate
{
"docs": [
{
"_source":{
"message": "~relevant message~"
}
}
]
}
after executing these 2 commands, I get a geoip field which consists of continent,city and region name, country and region iso code , latitude and longitude.
Now , I have elasticsearch index and index pattern for it. I need to add these fields to index so that they will be updated and I will be able to see them in the index pattern.
If you are saying that I should go with PUT Mapping API option then , if I am correct, by using PUT in dev tools it should work. This is how it should look like , I presume:
PUT /elasticsearch_index/_mappings/
{
"properties":{
~name of all fields I have listed above and their types~
}
The process above will add these fields to index, however , I don't think that any data will become visible ( by that I mean the fields that were created during pipeline simulation). Therefore, could you , please , write something similar to above to explain how can I save this pipeline to elasticsearch or whatever so that this template will treat all of these messages correctly and automatically do the mapping.
In other words, I can create these fields but how to ship the data or map it from my ip field automatically.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.