Geo-Point Mapping

Elastic Stack Elasticsearch
1 
mutate {
    add_field => {
      "location" => ["%{[location_lat]}","%{[location_lon]}"]
    }
  }

This is my mutate field in my conf file to create my geopoint mapping of the location_lat and location_lon value in the data. However, the value that is being created is simply strings with location_lon . I attempted to use this mapping but am confused when and where in Kibana I should utilize it.

PUT mongo_log_data
{
  "mappings": {
    "my_type": {
      "properties": {
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}
2

In Dev Tools, you need to run that before you put data into the index though. It's not a retrospective change.

3

Thank you!

filter {
mutate {
copy => { "_id" => "[@metadata][_id]"}
}
mutate{
    remove_field => ["_id"]
}
mutate{
    remove_field => ["image_url"]
}
mutate{
    remove_field => ["maintext"]
}

mutate {
    add_field => {
      "location" => ["%{[location_lat]}","%{[location_lon]}"]
    }
  }
}

I have this filter present in my logstash file but when I use this mapping, none of my data goes through. It is correctly processing the location as a geo point but no data is going through. I attempted to switch the date filter in Kibana but it still was empty. My logs are seemingly going through but I am not sure what I am doing incorrectly within this mapping.

PUT mongo_log_data
{
  "mappings": {
      "properties": { 
      "location": {
          "type": "geo_point"
        }
      }
    }
  }
4

I am not sure what you mean by this.

5

Does the mapping overwrite anything that I have in the logstash-conf file? I want all of those values from the conf file to be present and converted in that way but even my date_publish field has no values when I run this pipeline.

6

The mapping is applied to Elasticsearch, not to Logstash.

If things are not working as you expect, it'd be helpful to provide more information. Configs, sample logs, mappings etc etc.

8

I am able to utilize the logstash pipeline to import the data from MongoDB into the index mongo_log_data. Should I also use PUT mongo_log_data into the mapping if I want to set up the geo_point coordinates from the mongo_log_data index but want to maintain all of the automatically populated fields from the Logstash?

9

Yes. You need to put the mapping into Elasticsearch before you send the data from Logstash to Elasticsearch.

10

Thank you! When I don't use the mapping before I run the Logstash, I am able to get the data through to Kibana. However, when I do use the mapping and then run the Logstash, the location value is correct but none of my data is visible in Kibana and I am expanding the date range.

Screen Shot 2021-04-08 at 3.00.41 PM
Screen Shot 2021-04-08 at 3.00.41 PM2874×1404 275 KB

I am getting this error for some of the documents but the ones that do not have this error are still not visible.

[2021-04-08T14:55:28,221][WARN ][logstash.outputs.elasticsearch][main][ed6b6e6684787676524231407cc67cc26fdbbcbbf2d12a6024f121ffe25d7c79] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mongo_log_data", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x41099155>], :response=>{"index"=>{"_index"=>"mongo_log_data", "_type"=>"_doc", "_id"=>"DDHWsngBjzCHtQZt15dw", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [location] of type [geo_point]", "caused_by"=>{"type"=>"parse_exception", "reason"=>"unsupported symbol [%] in geohash [%{[location_lat]}]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"unsupported symbol [%] in geohash [%{[location_lat]}]"}}}}}}

11

Hi @caitlyn.carlisle

This is incorrect syntax...

Perhaps try something like ...

This is the correct of above you have extra []s in the right side

  add_field => {
     "location" => ["%{location_lat}","%{location_lon}"]
}

or

    rename => {
        "location_lon" => "[location][lon]"
        "location_lat" => "[location][lat]"
    }

or

    copy => {
        "location_lon" => "[location][lon]"
        "location_lat" => "[location][lat]"
    }
12

Thank you!

I implemented the first change but still got this error:

[2021-04-08T16:54:10,236][WARN ][logstash.outputs.elasticsearch][main][4ac1613603270b4eb26d5bbe0db2493e553a6a82019df81e86c19a58e579d342] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"mongo_log_data", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x7af918bc>], :response=>{"index"=>{"_index"=>"mongo_log_data", "_type"=>"_doc", "_id"=>"BPxDs3gBEV6jPE3FheCw", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [location] of type [geo_point]", "caused_by"=>{"type"=>"parse_exception", "reason"=>"unsupported symbol [%] in geohash [%{location_lat}]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"unsupported symbol [%] in geohash [%{location_lat}]"}}}}}}

13

Could there be something wrong with my mapping? I was able to get the pipeline to work with the initial code, even with the inaccurate code, it just wasn't working once I used this mapping?

PUT mongo_log_data
{
  "mappings": {
  "properties": { 
  "location": {
      "type": "geo_point"
    },
    
  "date_publish": {
      "type": "date"
    },
    
  "title": {
      "type": "keyword"
    },
    
  "url": {
      "type": "keyword"
    },
  
  "civic1_event_type": {
      "type": "keyword"
    },
    
  "RAI_event_type": {
      "type": "keyword"
    }
}
  }
}
14

The mapping looks good

Can you please post your entire logstash conf file exactly how it is we'll take a look. That error still indicates a syntax error so something's not right.

What do your lat and long Fields look like can you show a ruby debug output so we can see?

15 
input {
mongodb {
    uri => 'mongodb://caitc:k123%21@1527017/ml4p?authSource=ml4p&readPreference=primary&ssl=false'
    placeholder_db_dir => '/Users/caitlyncarlisle/desktop/logstash-7.12.0/bin'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'articles-2012-1'
    batch_size => 5000
}

}
filter {
mutate {
copy => { "_id" => "[@metadata][_id]"}
}

mutate{
remove_field => ["_id"]
}

mutate{
remove_field => ["image_url"]
}

mutate{
remove_field => ["maintext"]
}

mutate {
convert => { "civic1_event_type" => "string" }
}

mutate {
convert => { "RAI_event_type" => "string" }
}

mutate {
add_field => {
"location" => ["%{[location_lat]}","%{[location_lon]}"]
}
}

}
output {
stdout {
codec => rubydebug
}
elasticsearch {
action => "index"
index => "mongo_log_data"
hosts => ["localhost:9200"]
}
}

And here is an example:

{
                           "civic1_model_outputs_legalaction" => 10.96875,
                                             "RAI_event_type" => "-999",
                     "RAI_model_outputs_security_cooperation" => 3.552734375,
                                                   "language" => "en",
                                                  "title_rss" => "NULL",
                                "civic1_model_outputs_arrest" => -1.2744140625,
                        "civic1_model_outputs_defamationcase" => -0.0156707763671875,
                                                "date_modify" => "2021-02-10T06:36:39Z",
                                               "date_publish" => "2012-11-15T21:37:48Z",
                                          "civic1_event_type" => "legalaction",
              "RAI_model_outputs_bribery_economic_corruption" => 2.123046875,
                      "civic1_model_outputs_mobilizesecurity" => -0.312255859375,
                                        "language_translated" => "en",
                                        "maintext_translated" => "Secretary of State Hillary Clinton ion.\n(AFP)\nDaily newsletterReceive essential international news every morning Subscribe",
                                "civic1_model_outputs_censor" => -0.237060546875,
                        "civic1_model_outputs_violencelethal" => -0.07086181640625,
                           "civic1_model_outputs_changepower" => -0.67138671875,
                         "RAI_model_outputs_diplomatic_visit" => 1.37890625,
                             "RAI_model_outputs_surveillance" => -0.447265625,
                  "RAI_model_outputs_economic_aid_assistance" => -0.55908203125,
            "RAI_model_outputs_joint_security_force_exercise" => 0.07330322265625,
                                                       "host" => "Caitlyns-MBP.lan",
                 "RAI_model_outputs_trade_agreement_exchange" => -2.58984375,
                                                        "url" => "https://www.france24.com/en/20121115-clinton-testify-benghazi-attack-republicans",
         "RAI_model_outputs_intelligence_counterintelligence" => 3.244140625,
        "RAI_model_outputs_social_academic_cultural_activity" => -0.02435302734375,
              "RAI_model_outputs_media_campaign_intervention" => 0.056610107421875,
                 "RAI_model_outputs_trade_financial_sanction" => -1.279296875,
                      "RAI_model_outputs_diaspora_activation" => 0.1064453125,
                                                  "log_entry" => "{\"_id\"=>BSON::ObjectId('60237ef801e920023bd60068'), \"authors\"=>[\"News Wires\"], \"date_download\"=>2021-02-10 06:36:39 UTC, \"date_modify\"=>2021-02-10 06:36:39 UTC, \"date_publish\"=>2012-11-15 21:37:48 UTC, \"description\"=>\"Secretary of State Hillary Clinton agreed to testify in hearings on the September attack on the US mission in Benghazi, Libya on Thursday, as Republicans continued to fuel a feud over the handling of…\", \"filename\"=>\"en_20121115-clinton-testify-benghazi-attack-republicans_1612938999.html\", \"image_url\"=>\"https://s.france24.com/media/display/1eb82d40-09e0-11e9-ac2f-005056bff430/w:1280/p:16x9/clinton_24.jpg\", \"language\"=>\"en\", \"title\"=>\"Clinton to testify in Benghazi attack probe\", \"title_page\"=>\"Clinton to testify in Benghazi attack probe\", \"title_rss\"=>\"NULL\", \"source_domain\"=>\"france24.com\", \"maintext\"=>\"Secretary of State Hillary Clinton  Subscribe\", \"url\"=>\"https://www.france24.com/en/20121115-clinton-testify-benghazi-attack-republicans\", \"event_extracted\"=>0, \"include\"=>true, \"language_translated\"=>\"en\", \"maintext_translated\"=>\"Secretary of State Hillary Clinton ...\", \"title_translated\"=>\"Clinton to testify in Benghazi attack probe\", \"mordecai_locations\"=>{\"LBY\"=>[\"Benghazi\", \"Libya\"]}, \"civic1\"=>{\"event_type\"=>\"legalaction\", \"model_outputs\"=>{\"arrest\"=>-1.2744140625, \"censor\"=>-0.237060546875, \"changeelection\"=>-0.303466796875, \"changepower\"=>-0.67138671875, \"cooperate\"=>-0.673828125, \"coup\"=>-0.56640625, \"defamationcase\"=>-0.0156707763671875, \"disaster\"=>-0.342529296875, \"legalaction\"=>10.96875, \"legalchange\"=>-0.52587890625, \"martiallaw\"=>-0.97216796875, \"mobilizesecurity\"=>-0.312255859375, \"praise\"=>-1.857421875, \"protest\"=>-0.671875, \"purge\"=>-0.5283203125, \"raid\"=>0.006866455078125, \"threaten\"=>0.01409912109375, \"violencelethal\"=>-0.07086181640625, \"violencenonlethal\"=>-1.3564453125}}, \"month\"=>11, \"year\"=>2012, \"RAI\"=>{\"event_type\"=>\"-999\", \"model_outputs\"=>{\"-999\"=>-0.322265625, \"arms_transfer_security_aid_assistance\"=>-0.76171875, \"bribery_economic_corruption\"=>2.123046875, \"diaspora_activation\"=>0.1064453125, \"diplomatic_mediation\"=>1.263671875, \"diplomatic_recognition\"=>-1.4140625, \"diplomatic_sanction\"=>-0.50341796875, \"diplomatic_statement\"=>1.373046875, \"diplomatic_visit\"=>1.37890625, \"economic_aid_assistance\"=>-0.55908203125, \"intelligence_counterintelligence\"=>3.244140625, \"investment\"=>-0.25146484375, \"joint_security_force_exercise\"=>0.07330322265625, \"media_campaign_intervention\"=>0.056610107421875, \"social_academic_cultural_activity\"=>-0.02435302734375, \"political_process_policy_intervention\"=>0.48828125, \"professional_cultural_exchange\"=>-0.274658203125, \"security_cooperation\"=>3.552734375, \"security_force_facility_presence\"=>-0.26416015625, \"official_security_force_facility_presence\"=>-0.57958984375, \"surveillance\"=>-0.447265625, \"tech_transfer_investment\"=>1.078125, \"trade_agreement_exchange\"=>-2.58984375, \"trade_financial_sanction\"=>-1.279296875, \"transnational_organization_crime\"=>-1.1494140625}}}",
                                  "civic1_model_outputs_coup" => -0.56640625,
         "RAI_model_outputs_security_force_facility_presence" => -0.26416015625,
                                 "civic1_model_outputs_purge" => -0.5283203125,
    "RAI_model_outputs_arms_transfer_security_aid_assistance" => -0.76171875,
                           "civic1_model_outputs_legalchange" => -0.52587890625,
                                                   "@version" => "1",
                     "RAI_model_outputs_diplomatic_statement" => 1.373046875,
                            "civic1_model_outputs_martiallaw" => -0.97216796875,
           "RAI_model_outputs_professional_cultural_exchange" => -0.274658203125,
                     "civic1_model_outputs_violencenonlethal" => -1.3564453125,
                               "civic1_model_outputs_protest" => -0.671875,
                                  "civic1_model_outputs_raid" => 0.006866455078125,
                                              "source_domain" => "france24.com",
                              "civic1_model_outputs_disaster" => -0.342529296875,
"RAI_model_outputs_official_security_force_facility_presence" => -0.57958984375,
                                                   "filename" => "en_20121115-clinton-testify-benghazi-attack-republicans_1612938999.html",
                                                       "year" => 2012,
                                "civic1_model_outputs_praise" => -1.857421875,
    "RAI_model_outputs_political_process_policy_intervention" => 0.48828125,
         "RAI_model_outputs_transnational_organization_crime" => -1.1494140625,
                                            "event_extracted" => 0,
                                                 "@timestamp" => 2021-04-08T22:02:43.766Z,
                        "civic1_model_outputs_changeelection" => -0.303466796875,
                              "civic1_model_outputs_threaten" => 0.01409912109375,
                                                      "month" => 11,
                                                   "location" => [
    [0] "%{[location_lat]}",
    [1] "%{[location_lon]}"
],
                     "RAI_model_outputs_diplomatic_mediation" => 1.263671875,
                                     "RAI_model_outputs_-999" => -0.322265625,
                               "RAI_model_outputs_investment" => -0.25146484375,
                                              "date_download" => "2021-02-10T06:36:39Z",
                                                 "title_page" => "Clinton to testify in Benghazi attack probe",
                                                    "logdate" => "2021-02-10T06:36:40+00:00",
                             "civic1_model_outputs_cooperate" => -0.673828125,
                 "RAI_model_outputs_tech_transfer_investment" => 1.078125,
                                                   "mongo_id" => "60237ef801e920023bd60068",
                                                "description" => "Secretary of State Hillary Clinton agreed to testify in hearings on the September attack on the US mission in Benghazi, Libya on Thursday, as Republicans continued to fuel a feud over the handling of…",
                                           "title_translated" => "Clinton to testify in Benghazi attack probe",
                   "RAI_model_outputs_diplomatic_recognition" => -1.4140625,
                      "RAI_model_outputs_diplomatic_sanction" => -0.50341796875,
                                                      "title" => "Clinton to testify in Benghazi attack probe"

}

16

However when I run it without the mapping, I am able to get this in another example

{
                                                 "@timestamp" => 2021-04-08T22:13:17.262Z,
                        "civic1_model_outputs_changeelection" => -1.294921875,
                                               "location_lat" => 55.378051,
                                                  "log_entry" => "{\"_id\"=>BSON::ObjectId('60237f3e8cc71d1e208c9a2a'), \"authors\"=>[], \"date_download\"=>2021-02-10 06:37:49 UTC, \"date_modify\"=>2021-02-10 06:37:49 UTC, \"date_publish\"=>2012-11-26 15:48:38 UTC, \"description\"=>\"Advice on how you can protect your home or business from flooding, and limit the damage to your property.\", \"filename\"=>\"news_science-environment-20497598_1612939069.html\", \"image_url\"=>\"https://ichef.bbci.co.uk/news/1024/media/images/64390000/jpg/_64390370_floodhomepa.jpg\", \"language\"=>\"en\", \"title\"=>\"UK floods: How can you protect your home or business?\", \"title_page\"=>\"UK floods: How can you protect your home or business? - BBC News\", \"title_rss\"=>\"NULL\", \"source_domain\"=>\"bbc.com\", \"maintext\"=>\"UK floods: How can you protect your home or business? Published 26 November 2012\\nimage caption People, on average, have to wait up to nine months before they can return to their flood-damaged homes\\nMore than five-and-a-half millioed for your area.\", \"title_translated\"=>\"UK floods: How can you protect your home or business?\", \"mordecai_locations\"=>{\"GBR\"=>[\"United Kingdom of Great Britain and Northern Ireland\"]}, \"civic1\"=>{\"event_type\"=>\"disaster\", \"model_outputs\"=>{\"arrest\"=>-0.97900390625, \"censor\"=>-0.3583984375, \"changeelection\"=>-1.294921875, \"changepower\"=>-0.002994537353515625, \"cooperate\"=>-0.434814453125, \"coup\"=>-0.3828125, \"defamationcase\"=>-0.26025390625, \"disaster\"=>11.6015625, \"legalaction\"=>-0.489990234375, \"legalchange\"=>-1.2109375, \"martiallaw\"=>0.1685791015625, \"mobilizesecurity\"=>-1.3720703125, \"praise\"=>-0.833984375, \"protest\"=>-0.6376953125, \"purge\"=>-0.53662109375, \"raid\"=>-0.650390625, \"threaten\"=>0.1649169921875, \"violencelethal\"=>-0.0216827392578125, \"violencenonlethal\"=>-0.019256591796875}}, \"month\"=>11, \"year\"=>2012, \"RAI\"=>{\"event_type\"=>\"-999\", \"model_outputs\"=>{\"-999\"=>-0.5390625, \"arms_transfer_security_aid_assistance\"=>-1.603515625, \"bribery_economic_corruption\"=>-0.3955078125, \"diaspora_activation\"=>-0.0677490234375, \"diplomatic_mediation\"=>-0.71142578125, \"diplomatic_recognition\"=>-2.3828125, \"diplomatic_sanction\"=>-0.81982421875, \"diplomatic_statement\"=>3.83984375, \"diplomatic_visit\"=>-0.61865234375, \"economic_aid_assistance\"=>1.3427734375, \"intelligence_counterintelligence\"=>-0.80126953125, \"investment\"=>3.701171875, \"joint_security_force_exercise\"=>-1.6611328125, \"media_campaign_intervention\"=>-0.373779296875, \"social_academic_cultural_activity\"=>-0.4814453125, \"political_process_policy_intervention\"=>0.59326171875, \"professional_cultural_exchange\"=>-0.0151824951171875, \"security_cooperation\"=>-0.3154296875, \"security_force_facility_presence\"=>0.77880859375, \"official_security_force_facility_presence\"=>-0.84765625, \"surveillance\"=>2.044921875, \"tech_transfer_investment\"=>0.95458984375, \"trade_agreement_exchange\"=>-1.44140625, \"trade_financial_sanction\"=>-0.342041015625, \"transnational_organization_crime\"=>0.413330078125}}, \"location\"=>{\"lat\"=>55.378051, \"lon\"=>-3.435973}}",
                        "civic1_model_outputs_defamationcase" => -0.26025390625,
                             "civic1_model_outputs_cooperate" => -0.434814453125,
                              "civic1_model_outputs_disaster" => 11.6015625,
                     "RAI_model_outputs_diplomatic_statement" => 3.83984375,
                           "civic1_model_outputs_legalaction" => -0.489990234375,
                     "civic1_model_outputs_violencenonlethal" => -0.019256591796875,
                                                  "title_rss" => "NULL",
                  "RAI_model_outputs_economic_aid_assistance" => 1.3427734375,
                                            "event_extracted" => 0,
                     "RAI_model_outputs_security_cooperation" => -0.3154296875,
                                        "maintext_translated" => "UK floods: g service.\nIt also offers advice on how to develop a \"flood plan\" and what you and your neighbours need to do once a flood warning has been issued for your area.",
                     "RAI_model_outputs_diplomatic_mediation" => -0.71142578125,
                 "RAI_model_outputs_trade_financial_sanction" => -0.342041015625,
                        "civic1_model_outputs_violencelethal" => -0.0216827392578125,
                                               "location_lon" => -3.435973,
        "RAI_model_outputs_social_academic_cultural_activity" => -0.4814453125,
                                                      "title" => "UK floods: How can you protect your home or business?",
                           "civic1_model_outputs_legalchange" => -1.2109375,
                                                       "host" => "Caitlyns-MBP.lan",
    "RAI_model_outputs_arms_transfer_security_aid_assistance" => -1.603515625,
                                              "date_download" => "2021-02-10T06:37:49Z",
                                                    "logdate" => "2021-02-10T06:37:50+00:00",
                                "civic1_model_outputs_praise" => -0.833984375,
                               "RAI_model_outputs_investment" => 3.701171875,
         "RAI_model_outputs_security_force_facility_presence" => 0.77880859375,
                                                      "month" => 11,
                         "RAI_model_outputs_diplomatic_visit" => -0.61865234375,
            "RAI_model_outputs_joint_security_force_exercise" => -1.6611328125,
                                        "language_translated" => "en",
                                                   "mongo_id" => "60237f3e8cc71d1e208c9a2a",
                      "RAI_model_outputs_diaspora_activation" => -0.0677490234375,
                                     "RAI_model_outputs_-999" => -0.5390625,
                                                   "location" => [
    [0] "55.378051",
    [1] "-3.435973"
],
                                                   "language" => "en",
                                                       "year" => 2012,
                                              "source_domain" => "bbc.com",
                                                   "@version" => "1",
                      "civic1_model_outputs_mobilizesecurity" => -1.3720703125,
              "RAI_model_outputs_bribery_economic_corruption" => -0.3955078125,
                                 "civic1_model_outputs_purge" => -0.53662109375,
                                           "title_translated" => "UK floods: How can you protect your home or business?",
                              "civic1_model_outputs_threaten" => 0.1649169921875,
                 "RAI_model_outputs_tech_transfer_investment" => 0.95458984375,
    "RAI_model_outputs_political_process_policy_intervention" => 0.59326171875,
                                                "description" => "Advice on how you can protect your home or business from flooding, and limit the damage to your property.",
                                                 "title_page" => "UK floods: How can you protect your home or business? - BBC News",
                            "civic1_model_outputs_martiallaw" => 0.1685791015625,
                               "civic1_model_outputs_protest" => -0.6376953125,
                                "civic1_model_outputs_arrest" => -0.97900390625,
                                                "date_modify" => "2021-02-10T06:37:49Z",
                                "civic1_model_outputs_censor" => -0.3583984375,
           "RAI_model_outputs_professional_cultural_exchange" => -0.0151824951171875,
                                             "RAI_event_type" => "-999",
                                          "civic1_event_type" => "disaster",
                           "civic1_model_outputs_changepower" => -0.002994537353515625,
"RAI_model_outputs_official_security_force_facility_presence" => -0.84765625,
                                  "civic1_model_outputs_raid" => -0.650390625,
         "RAI_model_outputs_transnational_organization_crime" => 0.413330078125,
                                  "civic1_model_outputs_coup" => -0.3828125,
         "RAI_model_outputs_intelligence_counterintelligence" => -0.80126953125,
                 "RAI_model_outputs_trade_agreement_exchange" => -1.44140625,
              "RAI_model_outputs_media_campaign_intervention" => -0.373779296875,
                                               "date_publish" => "2012-11-26T15:48:38Z",
                                                        "url" => "https://www.bbc.com/news/science-environment-20497598",
                   "RAI_model_outputs_diplomatic_recognition" => -2.3828125,
                      "RAI_model_outputs_diplomatic_sanction" => -0.81982421875,
                             "RAI_model_outputs_surveillance" => 2.044921875,
                                                   "filename" => "news_science-environment-20497598_1612939069.html"

}

17

You still have the same syntax error

This has extra brackets... you did not paste in what I showed these []s ^ are extra

"location" => ["%{[location_lat]}","%{[location_lon]}"]
                  ^            ^      ^            ^

Should be

"location" => ["%{location_lat}","%{location_lon}"]
No Brackets       ^

The geo_point requires a specific type of data because of your syntax error you are trying to put a string in for location.lat and location.lon which are float s. This will work if you just clean up your syntax

18

Thank you! I changed the conf file but still getting this error

{
                         "RAI_model_outputs_diplomatic_visit" => -0.61865234375,
                             "civic1_model_outputs_cooperate" => -0.434814453125,
                                                      "month" => 11,
                                                  "title_rss" => "NULL",
                           "civic1_model_outputs_legalchange" => -1.2109375,
                     "RAI_model_outputs_diplomatic_statement" => 3.83984375,
                      "RAI_model_outputs_diplomatic_sanction" => -0.81982421875,
                                          "civic1_event_type" => "disaster",
                                                      "title" => "UK floods: How can you protect your home or business?",
                                  "civic1_model_outputs_coup" => -0.3828125,
                                                  "log_entry" => "{\"_id\"=>BSON::ObjectId('60237f3e8cc71d1e208c9a2a'), \"authors\"=>[], \"date_download\"=>2021-02-10 06:37:49 UTC, \"date_modify\"=>2021-02-10 06:37:49 UTC, \"date_publish\"=>2012-11-26 15:48:38 UTC, \"description\"=>\"Advice on how you can protect your home or business from flooding, and limit the damage to your property.\", \"filename\"=>\"news_science-environment-20497598_1612939069.html\", \"image_url\"=>\"https://ichef.bbci.co.uk/news/1024/media/images/64390000/jpg/_64390370_floodhomepa.jpg\", \"language\"=>\"en\", \"title\"=>\"UK floods: How can you protect your home or business?\", \"title_page\"=>\"UK floods: How can you protect your home or business? - BBC News\", \"title_rss\"=>\"NULL\", \"source_domain\"=>\"bbc.com\", \"maintext\"=>\"UK floods: How can you protect your home or business? Published 26 November 2012\\nimage caption People, on average, have to wait up to nine months before they can return to their flood-damaged homes\\nMore than five-and-a-half million properties in England and Wales are deemed to be at risk from flooding.\\nThis year hasd-water into the property?\\nThere are a number of routes in which floodwater can enter a propep a \\\"flood plan\\\" and what you and your neighbours need to do once a flood warning has been issued for your area.\", \"title_translated\"=>\"UK floods: How can you protect your home or business?\", \"mordecai_locations\"=>{\"GBR\"=>[\"United Kingdom of Great Britain and Northern Ireland\"]}, \"civic1\"=>{\"event_type\"=>\"disaster\", \"model_outputs\"=>{\"arrest\"=>-0.97900390625, \"censor\"=>-0.3583984375, \"changeelection\"=>-1.294921875, \"changepower\"=>-0.002994537353515625, \"cooperate\"=>-0.434814453125, \"coup\"=>-0.3828125, \"defamationcase\"=>-0.26025390625, \"disaster\"=>11.6015625, \"legalaction\"=>-0.489990234375, \"legalchange\"=>-1.2109375, \"martiallaw\"=>0.1685791015625, \"mobilizesecurity\"=>-1.3720703125, \"praise\"=>-0.833984375, \"protest\"=>-0.6376953125, \"purge\"=>-0.53662109375, \"raid\"=>-0.650390625, \"threaten\"=>0.1649169921875, \"violencelethal\"=>-0.0216827392578125, \"violencenonlethal\"=>-0.019256591796875}}, \"month\"=>11, \"year\"=>2012, \"RAI\"=>{\"event_type\"=>\"-999\", \"model_outputs\"=>{\"-999\"=>-0.5390625, \"arms_transfer_security_aid_assistance\"=>-1.603515625, \"bribery_economic_corruption\"=>-0.3955078125, \"diaspora_activation\"=>-0.0677490234375, \"diplomatic_mediation\"=>-0.71142578125, \"diplomatic_recognition\"=>-2.3828125, \"diplomatic_sanction\"=>-0.81982421875, \"diplomatic_statement\"=>3.83984375, \"diplomatic_visit\"=>-0.61865234375, \"economic_aid_assistance\"=>1.3427734375, \"intelligence_counterintelligence\"=>-0.80126953125, \"investment\"=>3.701171875, \"joint_security_force_exercise\"=>-1.6611328125, \"media_campaign_intervention\"=>-0.373779296875, \"social_academic_cultural_activity\"=>-0.4814453125, \"political_process_policy_intervention\"=>0.59326171875, \"professional_cultural_exchange\"=>-0.0151824951171875, \"security_cooperation\"=>-0.3154296875, \"security_force_facility_presence\"=>0.77880859375, \"official_security_force_facility_presence\"=>-0.84765625, \"surveillance\"=>2.044921875, \"tech_transfer_investment\"=>0.95458984375, \"trade_agreement_exchange\"=>-1.44140625, \"trade_financial_sanction\"=>-0.342041015625, \"transnational_organization_crime\"=>0.413330078125}}, \"location\"=>{\"lat\"=>55.378051, \"lon\"=>-3.435973}}",
                      "civic1_model_outputs_mobilizesecurity" => -1.3720703125,
                        "civic1_model_outputs_violencelethal" => -0.0216827392578125,
                 "RAI_model_outputs_trade_agreement_exchange" => -1.44140625,
                           "civic1_model_outputs_legalaction" => -0.489990234375,
                                "civic1_model_outputs_censor" => -0.3583984375,
                                                       "host" => "Caitlyns-MBP.lan",
                                                   "mongo_id" => "60237f3e8cc71d1e208c9a2a",
                        "civic1_model_outputs_changeelection" => -1.294921875,
                                            "event_extracted" => 0,
                        "civic1_model_outputs_defamationcase" => -0.26025390625,
                                                "description" => "Advice on how you can protect your home or business from flooding, and limit the damage to your property.",
                                     "RAI_model_outputs_-999" => -0.5390625,
        "RAI_model_outputs_social_academic_cultural_activity" => -0.4814453125,
                                        "language_translated" => "en",
    "RAI_model_outputs_political_process_policy_intervention" => 0.59326171875,
                 "RAI_model_outputs_tech_transfer_investment" => 0.95458984375,
                                               "location_lon" => -3.435973,
                                                        "url" => "https://www.bbc.com/news/science-environment-20497598",
                                                       "tags" => [
    [0] "_mutate_error"
],
                              "civic1_model_outputs_threaten" => 0.1649169921875,
"RAI_model_outputs_official_security_force_facility_presence" => -0.84765625,
                              "civic1_model_outputs_disaster" => 11.6015625,
                     "RAI_model_outputs_security_cooperation" => -0.3154296875,
                                                   "@version" => "1",
    "RAI_model_outputs_arms_transfer_security_aid_assistance" => -1.603515625,
            "RAI_model_outputs_joint_security_force_exercise" => -1.6611328125,
                                              "source_domain" => "bbc.com",
                      "RAI_model_outputs_diaspora_activation" => -0.0677490234375,
         "RAI_model_outputs_intelligence_counterintelligence" => -0.80126953125,
                                                   "language" => "en",
                  "RAI_model_outputs_economic_aid_assistance" => 1.3427734375,
                               "RAI_model_outputs_investment" => 3.701171875,
                                        "maintext_translated" => "UK floods: How can you protect your home or business? Published 26 November 2012\nimage caption People, on average, have to wait up to nine months before they can return to their flooent Agency's flood warning service.\nIt also offers advice on how to develop a \"flood plan\" and what you and your neighbours need to do once a flood warning has been issued for your area.",
                                           "title_translated" => "UK floods: How can you protect your home or business?",
                                "civic1_model_outputs_praise" => -0.833984375,
                                              "date_download" => "2021-02-10T06:37:49Z",
                                                 "title_page" => "UK floods: How can you protect your home or business? - BBC News",
                                                    "logdate" => "2021-02-10T06:37:50+00:00",
                     "civic1_model_outputs_violencenonlethal" => -0.019256591796875,
         "RAI_model_outputs_security_force_facility_presence" => 0.77880859375,
                            "civic1_model_outputs_martiallaw" => 0.1685791015625,
                                                       "year" => 2012,
                     "RAI_model_outputs_diplomatic_mediation" => -0.71142578125,
                                                   "location" => "55.378051",
                               "civic1_model_outputs_protest" => -0.6376953125,
                             "RAI_model_outputs_surveillance" => 2.044921875,
                                               "date_publish" => "2012-11-26T15:48:38Z",
              "RAI_model_outputs_bribery_economic_corruption" => -0.3955078125,
                                 "civic1_model_outputs_purge" => -0.53662109375,
                                                 "@timestamp" => 2021-04-08T22:27:07.629Z,
                   "RAI_model_outputs_diplomatic_recognition" => -2.3828125,
           "RAI_model_outputs_professional_cultural_exchange" => -0.0151824951171875,
                                               "location_lat" => 55.378051,
                                                "date_modify" => "2021-02-10T06:37:49Z",
                           "civic1_model_outputs_changepower" => -0.002994537353515625,
                                                   "filename" => "news_science-environment-20497598_1612939069.html",
                                             "RAI_event_type" => "-999",
              "RAI_model_outputs_media_campaign_intervention" => -0.373779296875,
                 "RAI_model_outputs_trade_financial_sanction" => -0.342041015625,
                                  "civic1_model_outputs_raid" => -0.650390625,
         "RAI_model_outputs_transnational_organization_crime" => 0.413330078125,
                                "civic1_model_outputs_arrest" => -0.97900390625

}

19

Should I be converting them to floats in my conf file?

20

This error? Not clear which mutate that is coming from.

They look like floats... so you shouldn't,

Did you clean up and reapply the mapping , then run logstash?

Why don't you try this version, that is what I like.

Lets try this, I know this works I don't use the other version.

copy => {
        "location_lon" => "[location][lon]"
        "location_lat" => "[location][lat]"
    }
21

Thank you so so much! This worked! I really appreciate your help!!

1 Like