Error parsing json {:source=>... Json::ParserError: Invalid numeric value: Leading zeroes not allowed at [Source: (byte[])

Hi,

I have an issue with my logstash configuration and I can't figure out why.

I have a custom logfile on a server which has filebeat installed. An example of the custom log

02/06/2019 11:51:14 Database Copy: DAG003-MDB159\QQQXM0030
02/06/2019 11:51:14 Server: QQQXM0030
02/06/2019 11:51:14 Activation Preference: 4
02/06/2019 11:51:14 Status: Healthy
02/06/2019 11:51:14 Copy Queue: 0
02/06/2019 11:51:14 Replay Queue: 14124
02/06/2019 11:51:14 Content Index: Healthy
02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030
02/06/2019 11:51:14 Replay lag is True
02/06/2019 11:51:14 Truncation lag is False
02/06/2019 11:51:14 ---- Processing database DAG003-MDB160
02/06/2019 11:51:14 DAG003-MDB160 has 4 copies
02/06/2019 11:51:14 Database Copy: DAG003-MDB160\QQQXM0023
02/06/2019 11:51:14 Server: QQQXM0023
02/06/2019 11:51:14 Activation Preference: 1
02/06/2019 11:51:14 Status: Mounted
02/06/2019 11:51:14 Copy Queue: 0
02/06/2019 11:51:14 Replay Queue: 0
02/06/2019 11:51:14 Content Index: Healthy
02/06/2019 11:51:14 Replay lag is False
02/06/2019 11:51:14 Truncation lag is False

Running filebeat on the server with multiline and outputting to a file.

filebeat.inputs:
- type: log
  enabled: true
  paths:
- "/opt/elastic/input/*.log"
  fields:
log_type: exchange
  fields_under_root: true
  tags: ["exchange"]

  ### Multiline options

  multiline.pattern: '------ Checking|Database Copy|---- Checking replication'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: 'Finished checking server|Truncation lag|DBLogReplayKeepingUp'

output.file:
  path: "/opt/elastic/"
  filename: output
  rotate_every_kb: 10000
  number_of_files: 7
  permissions: 0600

All works fine and the output seems to generate valid JSON. I can copy the output file to my logstash server and put it in a directory for logstash to ingest. All is well, my index is created in Elastic and everything looks good.

Here is a copy of my logstash pipeline with the grok filtering

input {
  #beats {
#id => "health_beats_plugin"
#port => 5044
  #}
  file {
path => "/etc/logstash/data/output"
start_position => "beginning"
sincedb_path => "/dev/null"
  }
}
filter {
  json {
  source => "message"
  target => "doc"
  }
  if [doc][log_type] == "exchange" {
mutate {
    gsub => [ "message", "\\n"," " ]
}
if "grokked" not in [tags] {
    grok {
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "message" => "%{DATESTAMP:logdate} ---- Checking replication health for %{HOSTNAME:hostname} %{DATESTAMP} ClusterService %{RESULT:clusterservice} %{DATESTAMP".....................}
        overwrite => [ "message" ]
        remove_field => ["message"]
        add_tag => ["replicationcheck", "grokked", "exchange"]
        tag_on_failure => [ ]
    }
}
date {
    match => [ "logdate", "MM/dd/yyyy HH:mm:ss" ]
    target => "@timestamp"
}
if "grokked" not in [tags] {
    drop { }
}
  }
}
output {
  if [doc][log_type] == "exchange" {
elasticsearch {...

The output in elastic index - looks good.

{
  "_index": "exchangehealth-2019.02.06",
  "_type": "doc",
  "_id": "2z0LFmkBzNgyfHMOj0Ey",
  "_version": 1,
  "_score": null,
  "_source": {
    "dbcopy": "DAG003-MDB159\\\\QQQXM0030",
    "@timestamp": "2019-02-06T11:51:14.000Z",
    "dbcopylag": "DAG003-MDB159",
    "hostname": "QQQXM0030",
    "replaylag": "True",
    "host": "qqqsubprd01",
    "@version": "1",
    "activationpreference": "4",
    "hostnamelag": "QQQXM0030",
    "logdate": "02/06/2019 11:51:14",
    "copyqueue": "0",
    "path": "/etc/logstash/data/outputzz",
    "contentindex": "Healthy",
    "truncationlag": "False",
    "doc": {
      "offset": 288911,
      "log": {
        "flags": [
          "multiline"
        ]
      },
      "@metadata": {
        "type": "doc",
        "version": "6.5.3",
        "beat": "filebeat"
      },
      "prospector": {
        "type": "log"
      },
      "@timestamp": "2019-02-21T00:58:03.284Z",
      "source": "/opt/elastic/input/exchangeserverhealth1.log",
      "host": {
        "name": "qqqelkrd01"
      },
      "beat": {
        "hostname": "qqqelkrd01",
        "version": "6.5.3",
        "name": "qqqelkrd01"
      },
      "input": {
        "type": "log"
      },
      "log_type": "exchange",
      "message": "02/06/2019 11:51:14 Database Copy: DAG003-MDB159\\QQQXM0030\n02/06/2019 11:51:14 Server: QQQXM0030\n02/06/2019 11:51:14 Activation Preference: 4\n02/06/2019 11:51:14 Status: Healthy\n02/06/2019 11:51:14 Copy Queue: 0\n02/06/2019 11:51:14 Replay Queue: 14124\n02/06/2019 11:51:14 Content Index: Healthy\n02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030\n02/06/2019 11:51:14 Replay lag is True\n02/06/2019 11:51:14 Truncation lag is False",
      "tags": [
        "UK",
        "ELK Node 2",
        "ELK",
        "exchange"
      ]
    },
    "status": "Healthy",
    "tags": [
      "dagcheck",
      "grokked",
      "exchange"
    ],
    "replayqueue": "14124"
  },
  "fields": {
    "@timestamp": [
      "2019-02-06T11:51:14.000Z"
    ],
    "doc.@timestamp": [
      "2019-02-21T00:58:03.284Z"
    ]
  },
  "sort": [
    1549453874000
  ]
}

If I switch the output from 'filebeat > file' to 'filebeat > logstash' and configure my logstash pipeline to listen for beats I get an Error (actually get lots of errors and the below is an example)

2019-02-22T13:37:42,839][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"02/06/2019 11:51:14 Database Copy: DAG003-MDB159\\QQQXM0030\n02/06/2019 11:51:14 Server: QQQXM0030\n02/06/2019 11:51:14 Activation Preference: 4\n02/06/2019 11:51:14 Status: Healthy\n02/06/2019 11:51:14 Copy Queue: 0\n02/06/2019 11:51:14 Replay Queue: 14124\n02/06/2019 11:51:14 Content Index: Healthy\n02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030\n02/06/2019 11:51:14 Replay lag is True\n02/06/2019 11:51:14 Truncation lag is False", :exception=>#<LogStash::Json::ParserError: Invalid numeric value: Leading zeroes not allowed
 at [Source: (byte[])"02/06/2019 11:51:14 Database Copy: DAG003-MDB159\QQQXM0030

Confused as it's the same filtering in logstash. Just changing the input to beats seems to break it??

Yes. In effect, the filebeat file output has a json codec. But if you use a logstash output the raw log file entries are sent as events. This is not JSON

"02/06/2019 11:51:14 Database Copy: DAG003-MDB159\QQQXM0030\n02/06/2019 11:51:14 Server: QQQXM0030\n02/06/2019 11:51:14 Activation Preference: 4\n02/06/2019 11:51:14 Status: Healthy\n02/06/2019 11:51:14 Copy Queue: 0\n02/06/2019 11:51:14 Replay Queue: 14124\n02/06/2019 11:51:14 Content Index: Healthy\n02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030\n02/06/2019 11:51:14 Replay lag is True\n02/06/2019 11:51:14 Truncation lag is False"

I suggest you remove the filter entirely and see what an event looks like using

output { stdout { codec => rubydebug } }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.