Converting string to ip datatype

Elastic Stack Logstash
1

Hi team ES...
Can anyone help me how to convert a string to ip datatype ,I have already indexed huge data... As I am new to ES ,i couldn't understand solutions to similar situation

sample data
//
current format
string ,date,string,sting,integer,string
// 10.8.1.18,01-08-2019 00:00:21,CURRENT,Idea/Vodafone,325 KB/s
10.8.1.18,01-08-2019 00:30:19,CURRENT,Idea/Vodafone,401 KB/s
10.8.1.18,01-08-2019 01:00:17,CURRENT,Idea/Vodafone,607 KB/s
10.8.1.18,01-08-2019 01:30:16,CURRENT,Idea/Vodafone,627 KB/s
10.8.1.18,01-08-2019 02:00:17,CURRENT,Idea/Vodafone,549 KB/s //

required format
ip ,date,string,sting,integer,string

my logstash config file content is

// input {
file {
path => "/home/bibin/logs///speed.csv"
max_open_files => 17000
start_position => "beginning"
sincedb_path => "/home/bibin/ALL/since_speed.db"
}
}
filter {
#10.8.1.18,05-07-2019 14:00:13,CURRENT,Idea/Vodafone,0 KB/s
#10.8.1.18,05-07-2019 14:30:22,CURRENT,Idea/Vodafone,254 KB/s
dissect {
mapping => {
"message" => "%{ip},%{occured_instant},%{time_slot},%{provider},%{speed} %{unit}"
}
}
date {
match => [ "occured_instant", "dd-MM-yyyy HH:mm:ss" ]
target => "occured_date"
}
mutate {
convert => {
"speed" => "integer"
}
}
}
output {
elasticsearch {
hosts => "192.168.0.12:9200"
index => "speed"
document_type => "speed_bbb"
}
stdout {}

2

Welcome @bibin, I have moved your post to the logstash category as it fits better there. They will be able to help you, thank you for posting!

Baz.

3

I would say this is an elasticsearch question. You need an index template that sets the type of the field on the document.

4

Hi Badger,
As per reference to es docs,I have created a new index "facebook" in my ES with datatype ip with following mapping and it was successful...
//
PUT /facebook/_mapping
{
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ip": {
"type": "ip"
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"my_field": {
"type": "text",
"fielddata": true
},
"occured_date": {
"type": "date"
},
"occured_instant": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"provider": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"speed": {
"type": "long"
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"time_slot": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"unit": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
//

and showed response as true...
//
{
"acknowledged" : true
}
//

But when I tried to index data into this new index ...
10.8.0.100,12-11-2019 01:00:41,CURRENT,Idea/Vodafone,73 KB/s
10.8.0.100,12-11-2019 01:30:12,CURRENT,Idea/Vodafone,0 KB/s

its is showing below mentioned error...

//
Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"facebook", :_type=>"face_docs", :routing=>nil}, #LogStash::Event:0x390714f], :response=>{"index"=>{"_index"=>"facebook", "_type"=>"face_docs", "_id"=>"aR6dh24BNAK6JMz4EyQN", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [facebook] as the final mapping would have more than 1 type: [_doc, face_docs]"}}}}
//

Can anyone help me why I am getting above error when trying to index documents with datatype ip to it..?

5

In the output of your elasticsearch remove the document_type that should work. Where as each index can have only one mapping type.

6

For more details on the removal of types see here.

7

Thanks chandu5565 for the help.It was successfully indexed by removing document_type from elasticsearch output. Hatsoff for the help.

I assume that as I have already specified mapping manually in the setting file,there is no need to specify document_type in the elasticsearch output part.

I am satisfied by the solution and was successful...!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.