Not giving correct column from translate pluging

Elastic Stack Logstash
1

two csv files are

hotel.csv
hotelcode,hotelname,citycode

CountryCityLive.csv
citycode,CityName,CountryRegionCode,CountryCode

taking all columns from hotel.csv and from CountryCityLive.csv only take CountryRegionCode.

translate {
destination => "[@metadata][citycode]"
dictionary_path => "/home/namalie/elk/travco/data/CountryCityLive.csv"
field => "citycode" }
dissect { mapping => { "[@metadata][citycode]" => "%{CountryRegionCode}" }}

But in data transferred in index has cityname data for CountryRegionCode.

and also following code not working if I want index with CityName and CountryRegionCode both

translate {
destination => "[@metadata][citycode]"
dictionary_path => "/home/namalie/elk/travco/data/CountryCityLive1.csv"
field => "citycode" }
dissect { mapping => { "[@metadata][citycode]" => "%{CityName};%{CountryRegionCode}" }}

2

If the logs are coming in line-by-line, and we can be sure of the number of values each time, we could probably also just use a dissect?

Eg:

input{...}
filter {
    dissect {
        mapping => {
            "message" => "%{citycode},%{CityName},%{CountryRegionCode},%{CountryCode}"
        }
    }
}

If you are definitely going to remove the other fields, and will never care about CountryRegionCode, we could also accumulate the useless fields:

input{...}
filter {
    dissect {
        mapping => {
            "message" => "%{useless},%{+useless},%{CountryRegionCode},%{+useless}"
        }
    }
    mutate {
        remove_field => [ "useless" ]
    }
}
3

You don't need to accumlate them, just don't name them.

"message" => "%{},%{},%{CountryRegionCode},%{}"
1 Like
4

thanks, I'll try this.

5

Thanks for the support, I'll try this as well

6

tried both but given solution didn't work
1)
translate {
destination => "[@metadata][statecode]"
dictionary_path => "/home/namalie/elk/travco/data/CountryCityLive1.csv"
field => "citycode" }
dissect { mapping => { "[@metadata][statecode]" => "%{useless},%{+useless},%{CountryRegionCode},%{+useless}" }}

didn't get anything

translate {
destination => "[@metadata][statecode]"
dictionary_path => "/home/namalie/elk/travco/data/CountryCityLive1.csv"
field => "citycode" }
dissect { mapping => { "message" => "%{},%{CityName},%{CountryRegionCode},%{}" }}

got fields as CityName and CountryRegionCode but the data are wrong from some other csv

translate {
destination => "[@metadata][statecode]"
dictionary_path => "/home/namalie/elk/travco/data/CountryCityLive1.csv"
field => "citycode" }
dissect { mapping => { "[@metadata][statecode]" => "%{useless},%{CityName},%{CountryRegionCode},%{+useless}" }}

didn't get anything

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.