# Custom fields visible in stdout, but not propagated to ElasticSearch

**URL:** <https://discuss.elastic.co/t/custom-fields-visible-in-stdout-but-not-propagated-to-elasticsearch/167157>\
**Category:** Logstash\
**Created:** [February 5, 2019, 4:12pm UTC](https://discuss.elastic.co/t/custom-fields-visible-in-stdout-but-not-propagated-to-elasticsearch/167157 "2019-02-05T16:12:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\
**Post date:** [February 5, 2019, 4:12pm UTC](https://discuss.elastic.co/t/custom-fields-visible-in-stdout-but-not-propagated-to-elasticsearch/167157/1 "2019-02-05T16:12:05Z")

</div>

Hi ELK community. I am quite newby, so I apologize in advance if the question is too simple.  
I have a configuration like this:

1. FileBeat (version 6.6.0) and LogStash (6.6.0) running on host A. For filtering, I use the plugin Grok. I have a few custom patterns in a mypatterns/ directory, generating a couple of new fields.

2. A second LogStash (vs 5.6.10) running on host B.

3. A ElasticSearch intance (5.6.10) running on host C.

To send data from LogStash A to LogStash B I am using the plugin Lumberjack. So, in host A I have something like this:

```
input {
    beats {
        port => "5044"
    }
}

filter {
    grok {
        match => { "message" => "%{PATTERN1:pattern1}"}
        add_field => { "event" => "starting" }
        break_on_match => false
        patterns_dir => ["/etc/logstash/patterns/starterlog"]
    }
    grok {
        match => { "message" => "%{PATTERN2:pattern2}"}
        add_field => { "event" => "finished" }
        break_on_match => false
        patterns_dir => ["/etc/logstash/patterns/starterlog"]
    }
}

output {
    lumberjack {
        codec => json
        hosts => "hostB"
        ssl_certificate => "/etc/logstash/certs/lumberjack.cert"
        port => 5555
    }
}

```

and the configuration in LogStash B is like this:

```
input {
    beats {
        codec => json
        port => 5555
        ssl => true
        ssl_certificate => "/etc/logstash/certs/lumberjack.cert"
        ssl_key => "/etc/logstash/certs/lumberjack.key"
    }
}

#output {
# stdout { codec => rubydebug }
#}

output {
    elasticsearch {
        hosts => ["hostC:9210"]
        index => "myindex"
    }
}

```

Here is the result.  
When I use output = stdout in LogStash B, everything seems to work fine. I see in the stdout the JSON docs, with those custom fields created in host A, as expected.  
However, when I use output = elasticsearch, the custom fields are not registered. I see the index is created, and it contains all the standard fields (like @timestamp, host, message, etc.) but none of mine (pattern1, pattern2, event).

Clearly is not a network problem, as the index is propagated fine from B to C. It has to be a configuration issue in the LogStash part.  
Any clue, hint, link to the exact piece of documentation explaining how to fix it, etc. is more than welcome.

Thanks a lot in advance.  
Cheers,  
Jose

---

<div class="post-metadata">

**Author:** ![jcaballero](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@jcaballero](https://discuss.elastic.co/u/jcaballero)\
**Post date:** [February 5, 2019, 8:14pm UTC](https://discuss.elastic.co/t/custom-fields-visible-in-stdout-but-not-propagated-to-elasticsearch/167157/2 "2019-02-05T20:14:34Z")

</div>

Update. If I change the setup, and I make the first LogStash to talk directly to ElasticSearch, I see exactly the same behavior. So it seems that using the lumberjack plugin is not the issue... But I still need to find out where is the problem exactly ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 5, 2019, 8:14pm UTC](https://discuss.elastic.co/t/custom-fields-visible-in-stdout-but-not-propagated-to-elasticsearch/167157/3 "2019-03-05T20:14:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.