# Logstash json input file only required fields to output

**URL:** <https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400>\
**Category:** Logstash\
**Created:** [September 6, 2023, 7:14am UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400 "2023-09-06T07:14:23Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 6, 2023, 7:14am UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/1 "2023-09-06T07:14:23Z")

</div>

I'm new in ELK & I have logs in JSON format. Below is the json sample log file. I want only item level array, others fields not required.  
**Sample logs**

```auto
{
"source": "mdm/pim",
  "topic": "pim-record-globalfields",
  "subject": "record",
  "eventType": "PIM.Export.FileCreated.Incremental",
  "eventTime": "2023-08-23T05:48:19Z",
  "data": {
    "item": [
      {
        "id": "373892",
        "crud": "Update",
        "fields": {
          "partid": "0004400000",
          "displaypartid": "0004400000",
          
        }
      },
      {
        "id": "373895",
        "crud": "Update",
        "fields": {
          "partid": "0006200000",
          "displaypartid": "0006200000",
         
        }
      }
    ]
  }
}

```

configuration file

```auto

input {
	file {
	
		path => "C:/narayan regal work/sample json files/sample2.json"
		 start_position => "beginning"
		sincedb_path => "NUL"
		codec => "json"
	}
}

filter {
   		
	json {
        source => "message"
      }
	mutate {
	 remove_field => [,"@version","source","host","log","@timestamp","event"]
	}
}

output {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "samplejsonforlogstash9"
			 document_type => "json"
			document_id => "%{id}"
		}
	
	stdout{}
}

```

Instead of removing fields, i want to get only required fields array

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 6, 2023, 8:17am UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/2 "2023-09-06T08:17:11Z")

</div>

I want only from item level array, others upper fields(source, topic to data) not required.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 6, 2023, 12:51pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/3 "2023-09-06T12:51:17Z")

</div>

You could try

```
    mutate { rename => { "[data][item]" => "item" } }
    prune { whitelist_names => ["item"] }

```

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 6, 2023, 3:50pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/4 "2023-09-06T15:50:12Z")

</div>

Thanks for quick response, it is working getting json output like this

```auto
[
    {
        "item": [
            {
                "fields": {
                    "partid": "0004400000",
                    "displaypartid": "0004400000"
                },
                "id": "373892",
                "crud": "Update"
            },
            {
                "fields": {
                    "partid": "0006200000",
                    "displaypartid": "0006200000"
                },
                "id": "373895",
                "crud": "Update"
            }
        ]
    }
]

```

But now i have to get output json like below. Please give advise how to achieve.

```auto
[
  {
    "fields": {
      "partid": "0004400000",
      "displaypartid": "0004400000"
    },
    "id": "373892",
    "crud": "Update"
  },
  {
    "fields": {
      "partid": "0006200000",
      "displaypartid": "0006200000"
    },
    "id": "373895",
    "crud": "Update"
  }
]

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 6, 2023, 4:14pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/5 "2023-09-06T16:14:14Z")

</div>

If you have an array in which the first entry is a hash that has a key called "item" that is an array of hashes and you want to move that to overwrite the outer array you could use

```
mutate { rename => { "[someField][0][item]" => "someField" } }

```

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 6, 2023, 4:40pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/6 "2023-09-06T16:40:59Z")

</div>

Can you please send me exact code, i could not able to understand. Thank you

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 6, 2023, 4:52pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/7 "2023-09-06T16:52:30Z")

</div>

I do not understand what your events look like. Can you show one using

```
output { stdout { codec => rubydebug } }

```

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 6, 2023, 4:59pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/8 "2023-09-06T16:59:34Z")

</div>

This is the output

```auto

{
    "item" => [
        [0] {
              "crud" => "Update",
                "id" => "373892",
            "fields" => {
                "displaypartid" => "0004400000",
                       "partid" => "0004400000"
            }
        },
        [1] {
              "crud" => "Update",
                "id" => "373895",
            "fields" => {
                "displaypartid" => "0006200000",
                       "partid" => "0006200000"
            }
        }
    ]
}

```

But i need in this format

```auto
[
  {
    "fields": {
      "partid": "0004400000",
      "displaypartid": "0004400000"
    },
    "id": "373892",
    "crud": "Update"
  },
  {
    "fields": {
      "partid": "0006200000",
      "displaypartid": "0006200000"
    },
    "id": "373895",
    "crud": "Update"
  }
]

```

This is the config file i am using after your suggestion

```auto
input {
	file {
		path => "C:/narayan regal work/sample json files/sample3.json"
		 start_position => "beginning"
		sincedb_path => "NUL"
		codec => "json"
	}
}

filter {
   		mutate { 
		   rename => { "[data][item]" => "item" }
		}
		prune { 
				whitelist_names => "item" 
		}
 }
output { stdout { codec => rubydebug } }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 6, 2023, 5:03pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/9 "2023-09-06T17:03:50Z")

</div>

> [@Narayan\_Rao](#):
>
> But i need in this format

I do not understand what you mean by that. It appears to be an array of hashes, but the array has to have a name, which is currently [item]. The field has to have a name.

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 8, 2023, 3:41am UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/10 "2023-09-08T03:41:40Z")

</div>

I want to insert only the elements inside "item" field to the elasticsearch each as an individual doc. I want to use their own id as elasticsearch doc id.  
The docs I want to insert into elasticsearch will be as follows:

```auto
[
  {
    "id": "373892",
    "crud": "Update",
    "fields": {
      "partid": "0004400000",
      "displaypartid": "0004400000"
    }
  },
  {
    "id": "373895",
    "crud": "Update",
    "fields": {
      "partid": "0006200000",
      "displaypartid": "0006200000"
    }
  }
]

```

JSON file which i am having right now

```auto
{
"source": "mdm/pim",
  "topic": "pim-record-globalfields",
  "subject": "record",
  "eventType": "PIM.Export.FileCreated.Incremental",
  "eventTime": "2023-08-23T05:48:19Z",
  "data": {
    "item": [
      {
        "id": "373892",
        "crud": "Update",
        "fields": {
          "partid": "0004400000",
          "displaypartid": "0004400000",
          
        }
      },
      {
        "id": "373895",
        "crud": "Update",
        "fields": {
          "partid": "0006200000",
          "displaypartid": "0006200000",
         
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 20, 2023, 12:51pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/11 "2023-09-20T12:51:11Z")

</div>

Thanks @Badger able to resolve the issue with your guidance, when json file is in single line.

```auto
input {
	
	file {
		path => "C:/narayan regal work/ELK data/pim-data/*.json"
		start_position => "beginning"
		sincedb_path => "NUL"
		codec => json
	}
}

filter {
      mutate { 
	   rename => { "[data][item]" => "item" }
	}
	split{
		field => "[item]"
	}

	prune { 
		whitelist_names => "item" 
	}
	ruby {
        code => '
			event.get("item").each { |k, v|
			event.set(k,v)
			}
			event.remove("item")
			'
		}
}

output {
	if "_rubyexception" not in [tags]{
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "samplejsonforlogstash5"
			 document_type => "json"
			document_id => "%{id}"
			doc_as_upsert => true
			action => "update"
		}
		 stdout{}
	}
}

```

---

<div class="post-metadata">

**Author:** ![Narayan\_Rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayan_rao/32/124566_2.png) [@Narayan\_Rao](https://discuss.elastic.co/u/Narayan_Rao)\
**Post date:** [September 20, 2023, 12:53pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/12 "2023-09-20T12:53:37Z")

</div>

@Badger How to convert multi line json file to single line json file

```auto

{
  "source": "logs",
  "data": {
    "item": [
      {
        "id": "101",
        "crud": "Create",
        "fields": {
          "partid": "101"
        },
        "logistics": {
          "weight": "6.10",
          "height": "4"
        }
      },
      {
        "id": "102",
        "crud": "Create",
        "fields": {
          "partid": "102"
        },
        "logistics": {
          "weight": "0.62",
          "height": "3"
        }
      },
      {
        "id": "103",
        "crud": "Create",
        "fields": {
          "partid": "103"
        },
        "logistics": {
          "weight": "4.88",
          "height": "2"
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 18, 2023, 12:54pm UTC](https://discuss.elastic.co/t/logstash-json-input-file-only-required-fields-to-output/342400/13 "2023-10-18T12:54:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.