Yet another logstash and json issue, unable to ingest a "basic" json

Elastic Stack Logstash
1

I have a dir with tons of subdirs with a json file like this:

 
{
    "id": "2234",
    "name": "Text",
    "url": "https://url.com",
    "param": "string",
    "last": "2021-07-06 20:05:49.458724",
    "url2": "url.com"
}

I've tried all things I'm able to search, actually I'm with this L config:

# Input section
input {
  file {
    type => "json"
	codec => multiline { pattern => "^Spalanzani" what => "previous" negate => true auto_flush_interval => 1 }

    path => "path/**/*.json"
    start_position => "beginning"
	sincedb_path => "path.log"
  }
}

#filter section
filter {
		  json {
			source => "message"
		  }
		mutate { 
		 remove_field => ["host", "@version",  "type", "path", "tags", "@timestamp"]				
		}	
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "myIndex"
    user => "usr" 
    password => "mypwd"
  }
    stdout {codec => rubydebug}
}

Don't know te reason but I'm sure that some time ago I was able to ingest a json file much more complex simply setting up type and condec to "json".... With previous config I get a message field filled with all fields, without last }, Why?
Any idea suggestion? Thanks.

2

Are you saying that with that example file you get

{    "id": "2234",    "name": "Text",    "url": "https://url.com",    "param": "string",    "last": "2021-07-06 20:05:49.458724",    "url2": "url.com"

in the message field?
3

Exactly!
I can't get tje point :frowning:
Why with this basic json E.L. can't ingest all data in each field inside the doc? What m I doing wrong?

4

I do not see anything wrong with your configuration. You could try increasing auto_flush_interval to see if that makes any difference but I very much doubt that it will.

5

The "problem" is that it inserts all the fields from the json file into the "message" field instead of creating a separate field within the elasticsearch doc for each field in the json file.
I have tried much more complex files that, once ingested by logstash to elasticsearch have had different fields. I don't know what changes from one to the other :S

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.