Logstash has duplicate timestamp field

Elastic Stack Logstash
1

I have some cassandra logs that use this custom pattern for logging

MILLISECOND (\d{3})
JAVALOGBACKTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}:%{SECOND},%{MILLISECOND}
CASS_BASE ^%{LOGLEVEL:level}  \[(%{DATA:process}:%{INT:threadId}|%{DATA:process})\] %{JAVALOGBACKTIMESTAMP:timestamp} %{WORD:java_file}.java:%{INT:line_number} -
FLUSHSIZE %{BASE10NUM}(KiB|GiB|MiB)
CASS_DEFAULT %{CASS_BASE} %{GREEDYDATA:message}

and it gets logged as the following in kibana

{
  "_index": "services-2019.05.30",
  "_type": "doc",
  "_id": "9PUtCWsBuizMRyGPkPJw",
  "_version": 1,
  "_score": null,
  "_source": {
    "source": "/opt/cassandra/logs/system.log",
    "java_file": "SSLFactory",
    "@version": "1",
    "index_prefix": "services",
    "logsource": "cassandra",
    "@timestamp": "2019-05-30T14:38:24.124Z",
    "timestamp": "2019-05-30 14:38:17,591",
    "beat": {
      "version": "6.2.1",
      "name": "myname",
      "hostname": "myhostname"
    },
    "tags": [
      "grokked",
      "leveled"
    ],
    "message": "Filtering out [TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] as it isn't supported by the socket",
    "line_number": "221",
    "level": "WARNING",
    "profiles": "myprofile",
    "offset": 6779433,
    "logtype": "service",
    "process": "epollEventLoopGroup-2-1"
  },
  "fields": {
    "@timestamp": [
      "2019-05-30T14:38:24.124Z"
    ]
  },
  "highlight": {
    "logsource": [
      "@kibana-highlighted-field@cassandra@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1559227104124
  ]
}

and my logstash config is the following

if "cassandra" in [logsource] {
    grok {
        patterns_dir => ["/etc/logstash/patterns"]
        match => ["message", "%{CASS_DEFAULT}"]
        overwrite => [ "message" ]
        add_tag => ["grokked"]
    }
  }

If i am setting the event field to be timestamp for the log timestamp, logstash should automatically set that as @timestamp right? Why is it not doing that, do i need a date filter for all logs to set @timestamp with the timestamp of the log file?

thanks

2

No, logstash will not do that automatically. You need to configure a date filter to do it.

3

thanks, how come in some of my cases logstash actually takes the timestamp field and sets is at @timestamp automatically.

4

for example i have a json log like this

	{"organization_id": "xxxxx", "account_id": "xxxxxx", "time": {"total": 60}, "@timestamp": "2019-05-30T03:00:46.418Z", "level": "INFO", "request": {"host": "timeseries-api-lb", "resource": "/organization/xxxxxxxx/series/frame/type/xxxxxx", "addr": "xxxx", "timespan": "1970-01-12T13:46:40.000000Z/1970-01-12T13:46:41.000000Z", "route": "/organization/*/account/*/series/frame/type/*", "chain_id": "xxxxx", "method": "GET", "agent": "java-sdk,2.5.0-dev.6.dev.101+549f3df;Mac OS X,10.14.1,x86_64;,,;java-sdk-test,BuildConfig.version", "size": ""}, "response": {"code": 200, "size": 708}, "submitter_id": 1}

using a json filter in logstash i see that kibana actually takes the timestamp from the json message and uses that as the @timestamp field

    },
    "@timestamp": "2019-05-30T03:00:46.418Z",
    "stream": "stdout",
    "input": {
      "type": "docker"
    },
    "kubernetes": {
      "node": {
        "name": "gke-piqdevkube01-physiq-00-4ca6871a-wdhm"
      },
      "labels": {
        "pod-template-hash": "566338998",
        "load-balancer-timeseries-api": "true",
        "name": "timeseries-api",
        "app": "timeseries"
      },
      "pod": {
        "name": "timeseries-api-9bb77dffd-xd9gr"
      },
      "namespace": "xxxx",
      "container": {
        "name": "timeseries"
      }
    },
    "beat": {
      "version": "6.3.2",
      "hostname": "filebeat-gvfcp",
      "name": "filebeat-gvfcp"
    },
    "organization_id": "xxxxxxxxxx",
    "account_id": "xxxxxxxxx"
  },
  "fields": {
    "@timestamp": [
      "2019-05-30T03:00:46.418Z"
    ]
  },

I see the same behavior when applying logstash patterns to certain logs as well so I'm confused as to how the @timestamp field is getting set automatically.

5

Your JSON does not have a timestamp field, it has an @timestamp field.

To be honest I am a little surprised that a json filter handles that as well as it does.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.