How to Ignore some messages from Kafka topic when consuming using Logstash

Elastic Stack Logstash
1

I have ten's of different messages in a kafka topic. I need to just consume messages that has only "America" in it. I have tried some suggestions available on the blog but no luck, could anyone help?

Flow is App -> Kafka topic -> Logstash consume pipeline and output to local ES.

Sample config:
input {
kafka {
bootstrap_servers => "broker:9093"
security_protocol => "SSL"
ssl_keystore_location => "/etc/logstash/keystore.jks"
ssl_keystore_password => "password"
ssl_truststore_location => "/etc/logstash/truststore.jks"
ssl_truststore_password => "password"
topics => ["topic1"]
codec => "json"
}
}
filter {
json{
source => "message"
}
}
output {
if "America" in [message] {
elasticsearch {
hosts => localhost
manage_template => false
index => "test"
}
}
}

2

Show us an example unwanted document. Copy/paste the raw JSON from Kibana's JSON tab.

3

{
"_index": "test",
"_type": "doc",
"_id": "XX",
"_version": 1,
"_score": null,
"_source": {
"meta": {
"message_type": "CREATE_ACCOUNT",
"create_timestamp": "2018-10-02T19:04:12.416Z",
"message_send_time": "2018-10-02T19:04:12.416Z",
"message_id": "XX"
},
"@version": "1",
"@timestamp": "2018-10-08T19:05:26.335Z",
"payload": {
"email": "XX@XX.com",
"last_name": "XX",
"first_name": "XX",
"value": XXOUT",
"id": "XX"
}
},
"fields": {
"@timestamp": [
"2018-10-08T19:05:26.335Z"
],
"meta.message_send_time": [
"2018-10-02T19:04:12.416Z"
],
"meta.create_timestamp": [
"2018-10-02T19:04:12.416Z"
]
},
"sort": [
1539025526335
]
}

4

So I have different message_type's. Ex: CREATE_ACCOUNT, DELETE_ACCOUNT
I would like to only consume CREATE_ACCOUNT.

5

I was trying:

filter {
if [@meta.message_type] == "DELETE_ACCOUNT" { drop{ } }
}

but no luck, any help is appreciated! TIA.

6

hey @magnusbaeck could you help when you get chance?

7

filter {
if [@meta.message_type] == "DELETE_ACCOUNT" { drop{ } }
}

Replace [@meta.message_type] with [meta][message_type].

1 Like
8

that did the trick! thanks much! you are the best!! @magnusbaeck

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.