# How does codec works in logstash

**URL:** <https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630>\
**Category:** Logstash\
**Created:** [October 19, 2017, 11:11pm UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630 "2017-10-19T23:11:47Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kenny123](https://avatars.discourse-cdn.com/v4/letter/k/7ab992/32.png) [@kenny123](https://discuss.elastic.co/u/kenny123)\
**Post date:** [October 19, 2017, 11:11pm UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630/1 "2017-10-19T23:11:48Z")

</div>

Hi,

I am learning ELK and implemented in test environment. logstash is reading the data from Kafka topic which is not in JSON format , but when I specified in logstash configuration that `codec => json` , I see that all the events are in JSON from logstash stdout. Can I please know how exactly codec works. If we specify codec in logstash, will the filters work ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 20, 2017, 4:47am UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630/2 "2017-10-20T04:47:18Z")

</div>

> I am learning ELK and implemented in test environment. logstash is reading the data from Kafka topic which is not in JSON format , but when I specified in logstash configuration that codec =\> json , I see that all the events are in JSON from logstash stdout.

I don't understand what you mean. What does your configuration look like? What events does Logstash produce?

> Can I please know how exactly codec works. If we specify codec in logstash, will the filters work ?

Yes, of course.

---

<div class="post-metadata">

**Author:** ![kenny123](https://avatars.discourse-cdn.com/v4/letter/k/7ab992/32.png) [@kenny123](https://discuss.elastic.co/u/kenny123)\
**Post date:** [October 20, 2017, 6:39pm UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630/3 "2017-10-20T18:39:28Z")

</div>

Thank you for the reply.. when I sent the data directly from file beats to logstash , in logstash output I could see field extractions but when data is sent from kafka to logstash without specifying codec in kafak input at logstash , I see the entire event . When the codec = json {} is specified in kafka input for logstash, I can field extractions again in logstash output. Can I please know without specifying codec for file beats everything is fine but for kafka, why we need to specify codec. how does it exactly works .

**filebeats to logstash**

```
"@timestamp" => 2016-09-19T21:26:02.887Z,
        "loglevel" => "INFO",
    "company_name" => "apache (additional instances)",
        "@version" => "1",
            "beat" => {
        "hostname" => "webserver"

```

**filebeats to kafka to logstash** ( without codec in logstash for kafka input )

`"message" => "{\"@timestamp\":\"2016-09-19T21:26:02.887Z\",\"beat\":{\"hostname\":\"Apps\"},\"fields\":{\"dc\":\"dc55\",\"env\":\"prod\",\"product\":\"database\",\"servertype\":\"web\"},\"input_type\":\"log\",\"message\":\"14:26:02,887 INFO [SQLExecutor@226cd970] [xx.xx.xx.xx] [12345] [apache]`

**filebeats to kafka to logstash** ( with codec in logstash for kafka input )

```
        type" => "webserver_log",
          "userid" => "admin",
      "@timestamp" => 016-09-19T21:26:02.887Z,
        "loglevel" => "INFO",
    "company_name" => "apache(additional instances)",
            "beat" => {
        "hostname" => "webserver",
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 23, 2017, 6:10am UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630/4 "2017-10-23T06:10:00Z")

</div>

It's very simple: If Filebeat is doing the JSON processing then Logstash obviously doesn't need to, but the Kafka input itself won't process the event as JSON (because it doesn't have to be JSON) so you have to explicitly ask Logstash to process it as JSON by choosing the json codec.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 20, 2017, 6:10am UTC](https://discuss.elastic.co/t/how-does-codec-works-in-logstash/104630/5 "2017-11-20T06:10:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.