Ingest json file

Elastic Stack Logstash
1

Currently I'm trying to ingest the following json file:

my_file.json

{
  "key_1": {
    "key_1-a": "value_a",
    "key_1-b": "value_b"
  },
  "key_2": {
    "key_2-a": "value_a",
    "key_2-b": "value_b"
  }
}

my_config.conf

input {
  file {

    codec => multiline
    {
        pattern => "^{"
        negate => true
        what => previous
    }

    path => "/etc/logstash/conf.d/my_file.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "json"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  stdout {  }
}

}

Currently i'm getting following error when i hit ctrl+c:

Error parsing json {:source=>"message", :raw=> ... " JSON content"
exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte)"

And also some error tags:

    [0] "multiline",
    [1] "_jsonparsefailure"

Any idea to solve it?
Thanks.

2

With the JSON that you show, that error does not occur. That suggests there is actually a problem with your JSON.

To consume an entire file as one event another option is to use a pattern that never matches and auto_flush.

codec => multiline { pattern => "^Spalanzani" what => "previous" negate => true auto_flush_interval => 1 }
1 Like
3

Thanks. Currently i'm getting following error:

[2019-04-05T16:57:51,322][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\n  \"Efectivo y equivalentes de efectivo (Nota 3)\": {\n    \"2016\": \"6,267,955\",\n    \"2017\": \"8,514,586\"\n  },\n  \"Deudores, neto (Nota 4)\": {\n    \"2016\": \"1,568,387\",\n    \"2017\": \"72,531\"\n  },\n  \"Total activos corrientes\": {\n    \"2016\": \"7,836,342\",\n    \"2017\": \"8,587,117\"\n  },\n  \"Cuentas por pagar (Nota 6)\": {\n    \"2016\": \"5,887,271\",\n    \"2017\": \"3,297,923\"\n  },\n  \"Impuestos y administraciones públicas (Nota 14)\": {\n    \"2016\": \"68,318\",\n    \"2017\": \"21,432\"\n  },\n  \"Beneficio a empleados (Nota 7)\": {\n    \"2016\": \"193,971\",\n    \"2017\": \"186,035\"\n  },\n  \"Total pasivos\": {\n    \"2016\": \"6,150,060\",\n    \"2017\": \"3,505,390\"\n  },\n  \"Superávit del año\": {\n    \"2016\": \"7,822,052\",\n    \"2017\": \"4,218,621\"\n  },\n  \"Total activos netos\": {\n    \"2016\": \"7,922,052\",\n    \"2017\": \"5,081,727\"\n  },\n  \"Total pasivos y saldos de activos netos\": {\n    \"2016\": \"14,072,112\",\n    \"2017\": \"8,587,117\"\n  },\n  \"Tarjeta Profesional\": {\n    \"2016\": \"61851-T Tarjeta\",\n    \"2017\": \"Profbsional 110141-T\"\n  }", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{
  "Efectivo y equivalentes de efectivo (Nota 3)": {
    "2016": "6,267,955",
    "2017": "8,514,586"
  },
....
  },
  "Impuestos y administraciones publicas (Nota 14)": {
    "2016": "68,318",
    "2017": "21,432"
  },
  "Beneficio a empleados (Nota 7"[truncated 491 bytes]; line: 45, column: 995]>}

Why this could happend?

4

If you look at your JSON it ends with

\"Profbsional 110141-T\"\n  }`

That closes out "Tarjeta Profesional", but does not close the overall object. Your JSON is missing a trailing }

5

Thanks for your help. But even using the following JSON with { and } start and ending still not working:

{
  "Efectivo y equivalentes de efectivo (Nota 3)": {
    "2016": "6,267,955",
    "2017": "8,514,586"
  },
  "Deudores, neto (Nota 4)": {
    "2016": "1,568,387",
    "2017": "72,531"
  },
  "Total activos corrientes": {
    "2016": "7,836,342",
    "2017": "8,587,117"
  },
  "Cuentas por pagar (Nota 6)": {
    "2016": "5,887,271",
    "2017": "3,297,923"
  },
  "Impuestos y administraciones públicas (Nota 14)": {
    "2016": "68,318",
    "2017": "21,432"
  },
  "Beneficio a empleados (Nota 7)": {
    "2016": "193,971",
    "2017": "186,035"
  },
  "Total pasivos": {
    "2016": "6,150,060",
    "2017": "3,505,390"
  },
  "Superávit del año": {
    "2016": "7,822,052",
    "2017": "4,218,621"
  },
  "Total activos netos": {
    "2016": "7,922,052",
    "2017": "5,081,727"
  },
  "Total pasivos y saldos de activos netos": {
    "2016": "14,072,112",
    "2017": "8,587,117"
  },
  "Tarjeta Profesional": {
    "2016": "61851-T Tarjeta",
    "2017": "Profbsional 110141-T"
  }
}

Same error:

[2019-04-05T18:13:52,075][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-04-05T18:13:53,954][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\n  \"Efectivo y equivalentes de efectivo (Nota 3)\": {\n    \"2016\": \"6,267,955\",\n    \"2017\": \"8,514,586\"\n  },\n  \"Deudores, neto (Nota 4)\": {\n    \"2016\": \"1,568,387\",\n    \"2017\": \"72,531\"\n  },\n  \"Total activos corrientes\": {\n    \"2016\": \"7,836,342\",\n    \"2017\": \"8,587,117\"\n  },\n  \"Cuentas por pagar (Nota 6)\": {\n    \"2016\": \"5,887,271\",\n    \"2017\": \"3,297,923\"\n  },\n  \"Impuestos y administraciones públicas (Nota 14)\": {\n    \"2016\": \"68,318\",\n    \"2017\": \"21,432\"\n  },\n  \"Beneficio a empleados (Nota 7)\": {\n    \"2016\": \"193,971\",\n    \"2017\": \"186,035\"\n  },\n  \"Total pasivos\": {\n    \"2016\": \"6,150,060\",\n    \"2017\": \"3,505,390\"\n  },\n  \"Superávit del año\": {\n    \"2016\": \"7,822,052\",\n    \"2017\": \"4,218,621\"\n  },\n  \"Total activos netos\": {\n    \"2016\": \"7,922,052\",\n    \"2017\": \"5,081,727\"\n  },\n  \"Total pasivos y saldos de activos netos\": {\n    \"2016\": \"14,072,112\",\n    \"2017\": \"8,587,117\"\n  },\n  \"Tarjeta Profesional\": {\n    \"2016\": \"61851-T Tarjeta\",\n    \"2017\": \"Profbsional 110141-T\"\n  }", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{
  "Efectivo y equivalentes de efectivo (Nota 3)": {
    "2016": "6,267,955",
    "2017": "8,514,586"
  },
  "Deudores, neto (Nota 4)": {
    "2016": "1,568,387",
    "2017": "72,531"
  },
  "Total activos corrientes": {
    "2016": "7,836,342",
    "2017": "8,587,117"
  },
  "Cuentas por pagar (Nota 6)": {
    "2016": "5,887,271",
    "2017": "3,297,923"
  },
  "Impuestos y administraciones públicas (Nota 14)": {
    "2016": "68,318",
    "2017": "21,432"
  },
  "Beneficio a empleados (Nota 7"[truncated 491 bytes]; line: 1, column: 1])
 at [Source: (byte[])"{
  "Efectivo y equivalentes de efectivo (Nota 3)": {
    "2016": "6,267,955",
    "2017": "8,514,586"
  },
  "Deudores, neto (Nota 4)": {
    "2016": "1,568,387",
    "2017": "72,531"
  },
  "Total activos corrientes": {
    "2016": "7,836,342",
    "2017": "8,587,117"
  },
  "Cuentas por pagar (Nota 6)": {
    "2016": "5,887,271",
    "2017": "3,297,923"
  },
  "Impuestos y administraciones públicas (Nota 14)": {
    "2016": "68,318",
    "2017": "21,432"
  },
  "Beneficio a empleados (Nota 7"[truncated 491 bytes]; line: 45, column: 995]>}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
       "message" => "{\n  \"Efectivo y equivalentes de efectivo (Nota 3)\": {\n    \"2016\": \"6,267,955\",\n    \"2017\": \"8,514,586\"\n  },\n  \"Deudores, neto (Nota 4)\": {\n    \"2016\": \"1,568,387\",\n    \"2017\": \"72,531\"\n  },\n  \"Total activos corrientes\": {\n    \"2016\": \"7,836,342\",\n    \"2017\": \"8,587,117\"\n  },\n  \"Cuentas por pagar (Nota 6)\": {\n    \"2016\": \"5,887,271\",\n    \"2017\": \"3,297,923\"\n  },\n  \"Impuestos y administraciones públicas (Nota 14)\": {\n    \"2016\": \"68,318\",\n    \"2017\": \"21,432\"\n  },\n  \"Beneficio a empleados (Nota 7)\": {\n    \"2016\": \"193,971\",\n    \"2017\": \"186,035\"\n  },\n  \"Total pasivos\": {\n    \"2016\": \"6,150,060\",\n    \"2017\": \"3,505,390\"\n  },\n  \"Superávit del año\": {\n    \"2016\": \"7,822,052\",\n    \"2017\": \"4,218,621\"\n  },\n  \"Total activos netos\": {\n    \"2016\": \"7,922,052\",\n    \"2017\": \"5,081,727\"\n  },\n  \"Total pasivos y saldos de activos netos\": {\n    \"2016\": \"14,072,112\",\n    \"2017\": \"8,587,117\"\n  },\n  \"Tarjeta Profesional\": {\n    \"2016\": \"61851-T Tarjeta\",\n    \"2017\": \"Profbsional 110141-T\"\n  }",
    "@timestamp" => 2019-04-05T23:13:53.751Z,
          "host" => "ofnanezn-AX370-Gaming-K5",
          "tags" => [
        [0] "multiline",
        [1] "_jsonparsefailure"
    ],
          "type" => "json",
          "path" => "/home/miguel/ocr/demo/finanzas_ocr1.json",
      "@version" => "1"
}
6

Both the error message and the "message" field that you are still missing the closing }

   "message" => "{\n [...] \"Tarjeta Profesional\": {\n    \"2016\": [...] }",

What does your input codec look like now?

7

Thanks for your help.

I don't understand why the message is different from the original json file.

I just passing exactly like this:

{
  "Efectivo y equivalentes de efectivo (Nota 3)": {
    "2016": "6,267,955",
    "2017": "8,514,586"
  },
  "Deudores, neto (Nota 4)": {
    "2016": "1,568,387",
    "2017": "72,531"
  },
  "Total activos corrientes": {
    "2016": "7,836,342",
    "2017": "8,587,117"
  },
  "Cuentas por pagar (Nota 6)": {
    "2016": "5,887,271",
    "2017": "3,297,923"
  },
  "Impuestos y administraciones públicas (Nota 14)": {
    "2016": "68,318",
    "2017": "21,432"
  },
  "Beneficio a empleados (Nota 7)": {
    "2016": "193,971",
    "2017": "186,035"
  },
  "Total pasivos": {
    "2016": "6,150,060",
    "2017": "3,505,390"
  },
  "Superávit del año": {
    "2016": "7,822,052",
    "2017": "4,218,621"
  },
  "Total activos netos": {
    "2016": "7,922,052",
    "2017": "5,081,727"
  },
  "Total pasivos y saldos de activos netos": {
    "2016": "14,072,112",
    "2017": "8,587,117"
  },
  "Tarjeta Profesional": {
    "2016": "61851-T Tarjeta",
    "2017": "Profbsional 110141-T"
  }
}

This is my input codec:

input {
  file {

    codec => multiline
    {
        pattern => "^Spalanzani"
        negate => true
        what => previous
        auto_flush_interval => 1
    }


    path => "/home/miguel/ocr/demo/finanzas_ocr1.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "json"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  stdout {}
}
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.