# Elasticsearch not saving all fields from logstash

**URL:** <https://discuss.elastic.co/t/elasticsearch-not-saving-all-fields-from-logstash/276957>\
**Category:** Logstash\
**Created:** [June 24, 2021, 3:27pm UTC](https://discuss.elastic.co/t/elasticsearch-not-saving-all-fields-from-logstash/276957 "2021-06-24T15:27:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mlassnig\_no](https://avatars.discourse-cdn.com/v4/letter/m/ce73a5/32.png) [@mlassnig\_no](https://discuss.elastic.co/u/mlassnig_no)\
**Post date:** [June 24, 2021, 3:27pm UTC](https://discuss.elastic.co/t/elasticsearch-not-saving-all-fields-from-logstash/276957/1 "2021-06-24T15:27:34Z")

</div>

I got following situation:  
Im sending logs from my servers to logstash using filebeat. Im running a logstash pipeline that looks like this:

```auto
input {
  beats {
    port => 5003
  }
}
filter {
    grok {
        match => { "message" => "(?m)%{DATESTAMP:timestamp} %{INT:process_id} %{WORD:level} %{NOTSPACE:db_name} %{USERNAME:name}: %{GREEDYDATA:log_message}"}
    }
    date {
        match => ["timestamp", "yy-MM-dd HH:mm:ss,SSS"]
        timezone => "Etc/UTC"
        target => "@timestamp"
    }
    mutate {
        remove_field => ["timestamp"]
        remove_field => ["message"]
    }

    if [log_message] =~ "^([0-9]{1,3}\.){3}[0-9]{1,3}" {
        grok {
            match => { "log_message" => "%{IPORHOST:request_clientip} %{USER:request_ident} %{USER:request_auth} \[%{GREEDYDATA:request_httpdate}\] \"(?:%{WORD:request_method} %{NOTSPACE:request_endpoint}(?: HTTP/%{NUMBER:request_httpversion})?|%{DATA:request_rawrequest})\" %{NUMBER:request_responsecode} (?:%{NUMBER:request_bytes}|-) %{GREEDYDATA:request_additionals}"}
        }
        date {
            match => ["request_httpdate", "dd/MMM/yyyy HH:mm:ss"]
            target => "request_httpdate"
        }
        mutate {
            add_field => ["type", "access"]
            remove_field => ["log_message"]
            rename => {
                "request_clientip" => "[request][clientip]"
                "request_ident" => "[request][ident]"
                "request_auth" => "[request][auth]"
                "request_httpdate" => "[request][httpdate]"
                "request_method" => "[request][method]"
                "request_endpoint" => "[request][endpoint]"
                "request_httpversion" => "[request][httpversion]"
                "request_rawrequest" => "[request][rawrequest]"
                "request_responsecode" => "[request][responsecode]"
                "request_bytes" => "[request][bytes]"
                "request_additionals" => "[request][additionals]"
            }
        }
    }
    else {
        mutate {
            add_field => ["type", "log"]
        }
    }

    mutate {
        remove_field => ["host"]
    }
}
output {
    if [type] == "log" {
        elasticsearch {
                hosts => "localhost:9200"
                data_stream => "true"
                data_stream_type => "logs"
                data_stream_dataset => "myapp"
                data_stream_namespace => "log"
            }
        stdout { codec => rubydebug }
    }
    if [type] == "access" {
        elasticsearch {
                hosts => "localhost:9200"
                data_stream => "true"
                data_stream_type => "logs"
                data_stream_dataset => "myapp"
                data_stream_namespace => "access"
            }
    }

}

```

My logstash stdout looks like this (exactly how i want it):

```auto
{
            "log" => {
          "file" => {
            "path" => "/var/log/myapp/applog.log"
        },
        "offset" => 74528864,
         "flags" => [
            [0] "multiline"
        ]
    },
            "ecs" => {
        "version" => "1.8.0"
    },
       "@version" => "1",
    "log_message" => "A \n very \n long \n multiline \n log",
          "agent" => {
                "type" => "filebeat",
            "hostname" => "myhost",
        "ephemeral_id" => "some id",
             "version" => "7.13.2",
                "name" => "myhost",
                  "id" => "another id"
    },
     "process_id" => "20104",
          "input" => {
        "type" => "log"
    },
          "level" => "TEST",
           "type" => "log",
        "db_name" => "my db name",
     "@timestamp" => 2021-05-20T06:02:18.478Z,
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
           "name" => "some field"
}

```

Up to this point everything works fine. The document is also written correctly to elasticsearch with the exception of the field "log\_message", which is always empty with multiline logs:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dcaa4ef6188e2707fcf123a54128bf917fdb5a3f.png)

As far as I can tell only multiline logs are affected. I also think the filebeat and logstash part works as I intended as the stdout returns the expected result.  
It might be worth noting im working with ES data streams as an output for logstash.  
Any ideas? Thanks in advance.

---

<div class="post-metadata">

**Author:** ![mlassnig\_no](https://avatars.discourse-cdn.com/v4/letter/m/ce73a5/32.png) [@mlassnig\_no](https://discuss.elastic.co/u/mlassnig_no)\
**Post date:** [June 25, 2021, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-not-saving-all-fields-from-logstash/276957/2 "2021-06-25T07:50:14Z")

</div>

My assumption was correct in thinking that elasticsearch was the cause of the problem.  
The "ignore above" setting for the mapping of the index caused long log lines to be dropped:

```auto
 "log_message": {
          "type": "keyword",
          "ignore_above": 1024
        }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 23, 2021, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-not-saving-all-fields-from-logstash/276957/3 "2021-07-23T07:50:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.