Logstash not showing field with null values

Elastic Stack Logstash
1

I am indexing elasticsearch via logstash but it is showing only document's fields with not null values.
Earlier I used to do this through transporter and it returns the null values as well. Also the mapping in both cases (transporter and logstash) is same.
Mongo --> Elasticsearch
eg - earlier response was [{"name":"xyz",city:null,"age":50,"additional_info":null}] now it is [{"name":"xyz","age":50}]

3

thanks for the quick response but I find that there is no output plugin like include_null_fields => true in logstash. can you suggest something further

4

thanks for the quick response but I find that there is no output plugin like include_null_fields => true in logstash. can you suggest something further

5

This is not correct, Logstash will not remove any field by default.

Not sure where this come from, but there is no such option in the elasticsearch output in Logstash.

Please share your entire Logstash configuration and a sample of your message, also are you combining Logstash with any module of Filebeat for example?

6 
input {
  mongodb {
    uri => "mongo_uri"
    placeholder_db_dir => "/usr/share/logstash/"
    placeholder_db_name => "logstash_sqlite.db"
    collection => "test"
    batch_size => 50000
  }
}

 

filter {
  mutate {
    remove_field => ["_id"]
  }
}
output {
  elasticsearch {
    hosts => ["http://ip:9200/"]
    index => "test"
  }
7 
{
                "_index": "test",
                "_id": "U2DOaIkBXYO636wKiXYo",
                "_score": 14.1151085,
                "_ignored": [
                    "log_entry.keyword"
                ],
                "_source": {
                    "mongo_id": "626943c4f7c76a6b553aa982",
                    "source_type": "__",
                    "img_link": "some source",
                    "logdate": "2022-04-27T13:23:16+00:00",
                    "uuid": "626943c4f7c76a6b553aa982",
                    "sources": "some source",
                    "full_name": "abc",
                    "designation": "xyz",
                    "profile_category": "__",
                    "log_entry": "{\"_id\"=>BSON::ObjectId('626943c4f7c76a6b553aa982'), \"created_date_time\"=>nil, \"profile_category\"=>\"abc\", \"Prefix\"=>nil, \"full_name\"=>\"abc\", \"first_name\"=>nil, \"last_name\"=>nil, \"native_name\"=>nil, \"birth_name\"=>nil, \"aliases\"=>[\"\"], \"gender\"=>nil, \"dob\"=>nil, \"age\"=>nil, \"birth_place\"=>nil, \"citizen_country\"=>[\"__\"], \"education\"=>nil, \"occupation\"=>nil, \"designation\"=>\"xyz\", \"positions_held\"=>nil, \"residence\"=>nil, \"sources\"=>\"something\", \"father\"=>nil, \"mother\"=>nil, \"siblings\"=>nil, \"spouse\"=>nil, \"children\"=>nil, \"relatives\"=>nil, \"wikipedia_link\"=>nil, \"img_link\"=>\"some link\"}",
                    "@timestamp": "2023-07-18T11:41:51.924906714Z",
                    "host": "2c2d8b232f8f",
                    "@version": "1"
                }
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.