Force insert double quotes in a field value

Elastic Stack Logstash
1

Hi all,

I have been trying to force wrap double quotes around a field value so it can be used in the query statements of the elasticsearch plugin.

the field in question is the network.community_id which includes a colon in it's value for instance 1:w+JDbLREk/O+VrHCp4fOHboKp10=. The problem is, when used in a query statement, the colon is being interpreted as a delimiter and therefore results in an error. To get around this, I have been trying to force add quotes around the field with gsub (target result "1:w+JDbLREk/O+VrHCp4fOHboKp10=") so the entire field can be interpreted as is. So far, no success. The quotes are being stripped of at the moment of the query.

Here is an example filter:

  filter {
       mutate {
           add_field => { "[@metadata][community_id]" => "%{[network][community_id]}" }

           # Wrap quotes around the temp community_id field
           gsub => [ '', '[@metadata][community_id]', '"[@metadata][community_id]"' ]
        }

        elasticsearch {
          hosts => "https://es-host:9200"
          index => "filebeat-*"
          query => "event.module:zeek AND network.community_id:%{[@metadata][community_id]}"
          fields => [["[zeek][files][sha1]","[zeek][files][sha1]"]]
          user => "es-user"
          password => "es-password"
          ca_file => "/etc/logstash/certificates/ca.crt"
        }
  }

And I get this error as a result:

[2021-05-26T21:53:23,209][WARN ][logstash.filters.elasticsearch][main][da92164e2cc4508fd39cbd10c9e3bd1855b867634e46a5010d9fe806786f8491] Failed to query elasticsearch for previous event {:index=>"filebeat-", :error=>"[400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [event.module:zeek AND network.community_id:1:w+JDbLREk/O+VrHCp4fOHboKp10=]","index_uuid":"c-wFXTIMSIqhRhf5GJERnQ","index":"filebeat-7.12.1-2021.05.06-000001"},{"type":"query_shard_exception","reason":"Failed to parse query [event.module:zeek AND network.community_id:1:w+JDbLREk/O+VrHCp4fOHboKp10=]","index_uuid":"Z_RtIU2iQ4eJDaI_zZI6UA","index":"filebeat-7.12.1-default-2021-2"},{"type":"query_shard_exception","reason":"Failed to parse query [event.module:zeek AND network.community_id:1:w+JDbLREk/O+VrHCp4fOHboKp10=]","index_uuid":"DTxKEyF1T0O4U1SAglveyQ","index":"filebeat-7.12.1-suricata-2021-2"},{"type":"parse_exception","reason":"parse_exception: Encountered \" \":\" \": \"\" at line 1, column 44.\nWas expecting one of:\n \n ...\n ...\n ...\n \"+\" ...\n \"-\" ...\n ...\n \"(\" ...\n \"\" ...\n \"^\" ...\n ...\n ...\n <FUZZY_SLOP> ...

Not sure how to get around this.

Thanks

2

add_field is executed after the gsub, so this needs to be split into two filters. And your gsub syntax is wrong

mutate { add_field => { "[@metadata][community_id]" => "%{[network][community_id]}" } }
mutate {
    gsub => [
        "[@metadata][community_id]", "^", '"',
        "[@metadata][community_id]", "$", '"'
    ]
}
3

@Badger You are the man! This did the trick.

Thanks for your help.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.