Append tag in Elasticsearch using Logstash

Elastic Stack Logstash
1

Hi all,

Quick question about tagging with Logstash. I am using the following filter:
filter {
mutate {
add_tag => [ "test" ]
}
}

It appears that the add_tag replaces any existing in tags field. Is there any way to append a tag?

I am using Logstash v6.2.1.

Thanks in advance.

2

add_tag appends tags. If you're not getting the tags you expect it's probably because of something else in your configuration.

3

Thank you Magnus for the quick response. Below is my full configuration and output from logstash. The result I receive is that the tags field of the record is reset with the value I provide in the add_tag option. As per below the tags field is not present in the event created by the jdbc input but exists in the document with the same id in elasticsearch. Could this be the reason?

Logstash Configuration:
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://ip:port/schema?zeroDateTimeBehavior=convertToNull"
jdbc_user => "user"
jdbc_password => "password"
jdbc_default_timezone => "UTC"
statement => "SELECT 'account' AS 'record_type', id AS 'record_id' FROM table WHERE TRUE and id = 1880713"
}
}

    filter {
      mutate {
        add_tag => [ "test" ]
      }
    }
	
    output {
    		elasticsearch 
    		{ 
    			hosts => ["localhost:9200"] 
    			index => "index"
    			document_type => "record"
    			document_id => "%{record_type}_%{record_id}"
                           action => "update"
    		}
    		stdout { codec => rubydebug }
    	}

Logstash output:
(0.007000s) SELECT
'account' AS 'record_type',
id 'record_id'
FROM table
WHERE TRUE
and id = 1880713

{
"@timestamp" => 2018-09-27T14:36:25.191Z,
"record_id" => 1880713,
"record_type" => "account",
"@version" => "1",
"tags" => [
[0] "test"
]

4

I don't understand. I've read through your configuration a number of times and the only place where you're adding a tag is the mutate filter adding the "test" tag. Naturally the only tag the document will have in the end is the "test" tag.

5

Thanks Magnus for your response. I will try to explain the scenario we're testing as I believe it wasn't clear from my comments:

  • Document with document_id: account_1880713 exists in Elasticsearch and has field tags populated
  • In Logstash we perform a query using jdbc input filter and depending on the result, we would like to add a tag to the existing document in Elasticsearch, using an update action
  • upon creating the event in Logstash we do not know the tags that exist already in Elasticsearch, What we are trying to achieve is to append to the existing list of the tags field.

Hope our use case is more clear now, let me know if you need any further info.

6

Oh, you're upserting the document. I think you have two options:

  • Use a scripted update. That's supported by the elasticsearch output but I'm not entirely sure you'll be able to use it in this case.
  • Use an elasticsearch filter to fetch the current contents of the document. You can then add your changes on top and push the document back to ES.
7

Many thanks Magnus, we'll test option 2.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.