# Logs not stored correctly in ES

**URL:** https://discuss.elastic.co/t/logs-not-stored-correctly-in-es/160903
**Category:** Logstash
**Created:** [December 14, 2018, 12:24pm UTC](https://discuss.elastic.co/t/logs-not-stored-correctly-in-es/160903 "2018-12-14T12:24:32Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![jeromepin](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@jeromepin](https://discuss.elastic.co/u/jeromepin)
#### Post date: [December 14, 2018, 12:24pm UTC](https://discuss.elastic.co/t/logs-not-stored-correctly-in-es/160903/1 "2018-12-14T12:24:33Z")

</div>

Hello,

Since a few days, my ELK+Fluentd stack stopped working as usual and I cannot find the source of my problem. I'm using :

- Logstash 6.5.3-1
- Filebeat version 6.4.0 (amd64), libbeat 6.4.0
- Elasticsearch 6.5

My logs are in bad shape in Kibana :

 ![38](https://us1.discourse-cdn.com/elastic/original/3X/2/2/22abd1e1c20ed9ac6a1e0f28aae43c43a2d97d05.png). Everything is JSON like. I don't remember when it changed but before, they were parsed correctly. Now, I cannot query them. You can easily understand the usefulness of a such system 😛

Here are my configs :

**Filebeat :**

```
- type: log
  paths:
    - /var/log/auth.log
    - /var/log/faillog
    - /var/log/secure
  exclude_files: [".gz$"]
  tags: ["auth"]
  fields_under_root: true
  processors:
  - add_locale: ~  

```

**Logstash :**

```
input {
    beats {
        port => "5044"
    }
}

filter {
    if "auth" in [tags] {
        grok {
            add_field => { "foo" => "bar" }
            match => {
                "message" => [
                    DELETED_BECAUSE_MESSAGE_WOULD_BE_TOO_LONG
                ]
            }
            pattern_definitions => {
                "GREEDYMULTILINE"=> "(.|\n)*"
            }
            # remove_field => "message"
        }
        date {
            match => ["[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
        geoip {
            source => "[system][auth][ssh][ip]"
            target => "[system][auth][ssh][geoip]"
        }
		json {
		    source => "system"
		}
    }
}

output {
    if "auth" in [tags] {
        stdout { }
        elasticsearch {
            hosts => "http://elasticsearch.example.lan:9200"
            index => "logs-auth-%{+YYYY.MM.dd}"
            ssl_certificate_verification => "false"
            template_overwrite => true
            template => "/etc/logstash/template.json"
            manage_template => "true"
        }
    }
}

```

As you can see, the filter is working well because the "foo" field is added and the geoip is done. Obviously, `grok` isn't working because it can't find the `message`.

The logstash logs are showing 2 things. First, I cannot force-parse some element (like `system` or `beat`) because it's not JSON :

```
   Error parsing json {:source=>"system", :raw=>{"auth"=>{"hostname"=>"REDACTED", "ssh"=>{"ip"=>"REDACTED", "geoip"=>{"country_code2"=>"SG", "region_code"=>"01", "latitude"=>REDACTED, "continent_code"=>"AS", "ip"=>"REDACTED", "longitude"=>REDACTED, "country_name"=>"Singapore", "country_code3"=>"SG", "city_name"=>"Singapore", "region_name"=>"Central Singapore Community Development Council", "timezone"=>"Asia/Singapore", "location"=>{"lon"=>REDACTED, "lat"=>REDACTED}}}, "timestamp"=>"Dec 14 12:55:02", "pid"=>"20333"}}, :exception=>java.lang.ClassCastException: org.jruby.RubyHash cannot be cast to org.jruby.RubyIO}

```

And second, the log is sent properly by filebeat :

```
{
          "beat" => {
        "hostname" => "REDACTED",
        "timezone" => "+01:00",
         "version" => "6.5.3",
            "name" => "REDACTED"
    },
       "message" => "Dec 14 12:55:02 REDACTED sshd[20333]: Disconnected from REDACTED port 48204 [preauth]",
         "input" => {
        "type" => "log"
    },
           "foo" => "bar",
        "source" => "/var/log/auth.log",
          "host" => {
        "name" => "REDACTED"
    },
        "fields" => {
        "cluster" => "dmz",
            "env" => "int",
         "region" => "REDACTED"
    },
        "offset" => 1031460,
      "@version" => "1",
    "prospector" => {
        "type" => "log"
    },
    "@timestamp" => 2018-12-14T11:55:02.000Z,
        "system" => {
        "auth" => {
             "hostname" => "REDACTED",
                  "ssh" => {
                   "ip" => "REDACTED",
                "geoip" => {
                     "country_code2" => "SG",
                       "region_code" => "01",
                          "latitude" => REDACTED,
                    "continent_code" => "AS",
                                "ip" => "REDACTED",
                         "longitude" => REDACTED,
                      "country_name" => "Singapore",
                     "country_code3" => "SG",
                         "city_name" => "Singapore",
                       "region_name" => "Central Singapore Community Development Council",
                          "timezone" => "Asia/Singapore",
                          "location" => {
                        "lon" => REDACTED,
                        "lat" => REDACTED
                    }
                }
            },
            "timestamp" => "Dec 14 12:55:02",
                  "pid" => "20333"
        }
    },
          "tags" => [
        [0] "auth",
        [1] "beats_input_codec_plain_applied",
        [2] "_jsonparsefailure"
    ]
} 

```

So I cannot understand why the objects are not parsed into ES.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 11, 2019, 12:24pm UTC](https://discuss.elastic.co/t/logs-not-stored-correctly-in-es/160903/2 "2019-01-11T12:24:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.