# Issue with logsatsh

**URL:** <https://discuss.elastic.co/t/issue-with-logsatsh/330227>\
**Category:** Logstash\
**Created:** [April 18, 2023, 1:55pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227 "2023-04-18T13:55:10Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\
**Post date:** [April 24, 2023, 10:48am UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/23 "2023-04-24T10:48:25Z")

</div>

you can find here

 ![dateformat](https://us1.discourse-cdn.com/elastic/original/3X/2/2/2275720d17310d2df4d856a0e8735b606e1c6550.png)

your problem is that your data contains 2 formats of date

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 24, 2023, 1:52pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/24 "2023-04-24T13:52:47Z")

</div>

Found that the problem is my date format. I create a new file which only contains the date in json format. And I have a json parse failure  
Here is the content of the file : {"creation\_date": "2023-01-04T15:30:28"}

And here is my config file :  
input {  
file {  
codec =\> "json"  
path =\> "/testELK/exemple.json"  
start\_position =\> "beginning"  
type =\> "json"  
}  
}  
filter {  
json {  
source =\> "message"  
remove\_field =\> ["message"]  
}  
date {  
match =\> ["creation\_date", "yyyy-MM-dd'T'HH:mm:ss.SSS"]  
}  
}  
output {  
elasticsearch {  
hosts =\> ["[http://10.69.115.190:9200](http://10.69.115.190:9200)"]  
index =\> "data1"  
}  
stdout {  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\
**Post date:** [April 24, 2023, 2:09pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/25 "2023-04-24T14:09:43Z")

</div>

the part of

```auto
remove_field => ["message"]

```

must go into mutate not into json filter plugin

```auto
mutate {
  remove_field => ["message"]
}

```

it may be like this

```auto
input {
  file {
    codec => "json"
    path => "/testELK/exemple.json"
    start_position => "beginning"
    type => "json"
  }
}
filter {
  json {
    source => "message"
  }
  date {
    match => ["creation_date", "yyyy-MM-dd'T'HH:mm:ss.SSS"]
  }
  mutate {
    remove_field => ["message"]
  }
}
output {
  elasticsearch {
   hosts => ["http://10.69.115.190:9200"]
   index => "data1"
  }
  stdout {
   codec => rubydebug
  }
}

```

here when you are writin please use the simbols on top to format the text to see them bether

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 25, 2023, 12:41pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/26 "2023-04-25T12:41:48Z")

</div>

Hello, I am trying to parse this JSON with Logstash.

```auto
{"creation_date": "2023/01/04", "vulnerabilities": [{"count": 1, "score": null, "vuln_index": 414, "plugin_name": "WordPad History", "severity": 0, "vpr_score": null, "plugin_id": 92438, "severity_index": 0, "cpe": "cpe:/o:microsoft:windows", "offline": false, "plugin_family": "Windows", "snoozed": 0}, {"count": 1, "score": null, "vuln_index": 396, "plugin_name": "Windows Defender Installed", "severity": 0, "vpr_score": null, "plugin_id": 131023, "severity_index": 1, "cpe": "cpe:/a:microsoft:windows_defender", "offline": false, "plugin_family": "Windows", "snoozed": 0}, {"count": 1, "score": null, "vuln_index": 333, "plugin_name": "RDP Screenshot", "severity": 0, "vpr_score": null, "plugin_id": 66173, "severity_index": 2, "cpe": "cpe:/o:microsoft:windows", "offline": false, "plugin_family": "General", "snoozed": 0}]}

```

Everything works fine, but when I add another object to my JSON, it fails. I don't understand why. ( I add another object in vulnerabilities key).

I got this error :

```auto
[ERROR] 2023-04-25 14:40:13.055 [[main]<file] json - JSON parse error, original data now in message field {:message=>"Unrecognized token 'count': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"count\": 1, \"score\": null, \"vuln_index\": 319, \"plugin_name\": \"NetBIOS Multiple IP Address Enumeration\", \"severity\": 0, \"vpr_score\": null, \"plugin_id\": 43815, \"severity_index\": 3, \"cpe\": null, \"offline\": false, \"plugin_family\": \"Windows\", \"snoozed\": 0}]}\"; line: 1, column: 6]", :exception=>LogStash::Json::ParserError, :data=>"count\": 1, \"score\": null, \"vuln_index\": 319, \"plugin_name\": \"NetBIOS Multiple IP Address Enumeration\", \"severity\": 0, \"vpr_score\": null, \"plugin_id\": 43815, \"severity_index\": 3, \"cpe\": null, \"offline\": false, \"plugin_family\": \"Windows\", \"snoozed\": 0}]}"}
[WARN] 2023-04-25 14:40:13.206 [[main]>worker0] json - Error parsing json {:source=>"message", :raw=>"count\": 1, \"score\": null, \"vuln_index\": 319, \"plugin_name\": \"NetBIOS Multiple IP Address Enumeration\", \"severity\": 0, \"vpr_score\": null, \"plugin_id\": 43815, \"severity_index\": 3, \"cpe\": null, \"offline\": false, \"plugin_family\": \"Windows\", \"snoozed\": 0}]}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'count': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"count": 1, "score": null, "vuln_index": 319, "plugin_name": "NetBIOS Multiple IP Address Enumeration", "severity": 0, "vpr_score": null, "plugin_id": 43815, "severity_index": 3, "cpe": null, "offline": false, "plugin_family": "Windows", "snoozed": 0}]}"; line: 1, column: 7]>}
{
          "type" => "json",
          "path" => "/testELK/exemple.json",
      "@version" => "1",
          "host" => "lnessus02v",
    "@timestamp" => 2023-04-25T12:40:13.066Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 25, 2023, 12:59pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/27 "2023-04-25T12:59:24Z")

</div>

> [@cperzrt10](#):
>
> the part of
> 
> ```auto
> remove_field => ["message"]
> 
> ```
> 
> must go into mutate not into json filter plugin

That is not correct. When the common remove\_field option is in the json filter the removal of the [message] field only occurs if the filter successfully parses the JSON. If it fails then the [message] filter is retained and you can examine it to see why it failed to parse.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 25, 2023, 1:02pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/28 "2023-04-25T13:02:30Z")

</div>

Hello can you help me about my issue?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 25, 2023, 1:07pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/29 "2023-04-25T13:07:29Z")

</div>

The string that it is trying to parse starts with `count"`, not `{"creation_date":`. That suggests that when you added another vulnerability you got the JSON structure wrong.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 25, 2023, 1:18pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/30 "2023-04-25T13:18:00Z")

</div>

no the structure is not wrong. When I add another vulnerability my json look like this :

```auto
{"creation_date": "2023/01/04", "vulnerabilities": [{"count": 1, "score": null, "vuln_index": 414, "plugin_name": "WordPad History", "severity": 0, "vpr_score": null, "plugin_id": 92438, "severity_index": 0, "cpe": "cpe:/o:microsoft:windows", "offline": false, "plugin_family": "Windows", "snoozed": 0}, {"count": 1, "score": null, "vuln_index": 396, "plugin_name": "Windows Defender Installed", "severity": 0, "vpr_score": null, "plugin_id": 131023, "severity_index": 1, "cpe": "cpe:/a:microsoft:windows_defender", "offline": false, "plugin_family": "Windows", "snoozed": 0}, {"count": 1, "score": null, "vuln_index": 333, "plugin_name": "RDP Screenshot", "severity": 0, "vpr_score": null, "plugin_id": 66173, "severity_index": 2, "cpe": "cpe:/o:microsoft:windows", "offline": false, "plugin_family": "General", "snoozed": 0},{"count": 1, "score": null, "vuln_index": 319, "plugin_name": "NetBIOS Multiple IP Address Enumeration", "severity": 0, "vpr_score": null, "plugin_id": 43815, "severity_index": 3, "cpe": null, "offline": false, "plugin_family": "Windows", "snoozed": 0}]}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 25, 2023, 1:36pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/31 "2023-04-25T13:36:45Z")

</div>

A json filter will parse that producing an array of four vulnerabilities ending in

```
    [3] {
            "vuln_index" => 319,
                 "count" => 1,
                 "score" => nil,
        "severity_index" => 3,
                   "cpe" => nil,
             "plugin_id" => 43815,
           "plugin_name" => "NetBIOS Multiple IP Address Enumeration",
         "plugin_family" => "Windows",
               "offline" => false,
             "vpr_score" => nil,
               "snoozed" => 0,
              "severity" => 0
    }

```

I do not think your data looks the way you think it looks.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 25, 2023, 1:41pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/32 "2023-04-25T13:41:02Z")

</div>

here is my data :

```auto
{"creation_date": "2023/01/04", "vulnerabilities": [{"count": 1, "score": null, "vuln_index": 414, "plugin_name": "WordPad History", "severity": 0, "vpr_score": null, "plugin_id": 92438, "severity_index": 0, "cpe": "cpe:/o:microsoft:windows", "offline": false, "plugin_family": "Windows", "snoozed": 0}, {"count": 1, "score": null, "vuln_index": 396, "plugin_name": "Windows Defender Installed", "severity": 0, "vpr_score": null, "plugin_id": 131023, "severity_index": 1, "cpe": "cpe:/a:microsoft:windows_defender", "offline": false, "plugin_family": "Windows", "snoozed": 0}, {"count": 1, "score": null, "vuln_index": 333, "plugin_name": "RDP Screenshot", "severity": 0, "vpr_score": null, "plugin_id": 66173, "severity_index": 2, "cpe": "cpe:/o:microsoft:windows", "offline": false, "plugin_family": "General", "snoozed": 0},{"count": 1, "score": null, "vuln_index": 319, "plugin_name": "NetBIOS Multiple IP Address Enumeration", "severity": 0, "vpr_score": null, "plugin_id": 43815, "severity_index": 3, "cpe": null, "offline": false, "plugin_family": "Windows", "snoozed": 0}]}

```

here is my confif file :

```auto
input {
   file {
    path => "/testELK/exemple.json"
    codec => "json"
    start_position => "beginning"
    type => "json"
  }
}

filter {
    json {
     source => "message"
   }
   mutate {
    remove_field => ["message"]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "data7"
  }
  stdout {
    codec => rubydebug
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 25, 2023, 3:01pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/33 "2023-04-25T15:01:27Z")

</div>

If you are using a json codec then there is no reason to use a json filter. When you use a json codec there will not be a field called [message] unless parsing the JSON fails. If it fails in the codec then [message] will get created and the json filter will get the same parse failure as the codec.

I suggest you remove the json and mutate filters and see what the [message] field looks like when it gets to elasticsearch. I am sure it will not look the way you expect.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 25, 2023, 3:08pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/34 "2023-04-25T15:08:40Z")

</div>

When I remove the filter I got this

```auto
[ERROR] 2023-04-25 17:05:59.794 [[main]<file] json - JSON parse error, original data now in message field {:message=>"Unrecognized token 'count': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"count\": 1, \"score\": null, \"vuln_index\": 300, \"plugin_name\": \"Microsoft Windows SMB Registry : Winlogon Cached Password Weakness\", \"severity\": 0, \"vpr_score\": null, \"plugin_id\": 11457, \"severity_index\": 4, \"cpe\": null, \"offline\": false, \"plugin_family\": \"Windows\", \"snoozed\": 0}]}\"; line: 1, column: 6]", :exception=>LogStash::Json::ParserError, :data=>"count\": 1, \"score\": null, \"vuln_index\": 300, \"plugin_name\": \"Microsoft Windows SMB Registry : Winlogon Cached Password Weakness\", \"severity\": 0, \"vpr_score\": null, \"plugin_id\": 11457, \"severity_index\": 4, \"cpe\": null, \"offline\": false, \"plugin_family\": \"Windows\", \"snoozed\": 0}]}"}
{
          "type" => "json",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => " ******",
       "message" => "count\": 1, \"score\": null, \"vuln_index\": 300, \"plugin_name\": \"Microsoft Windows SMB Registry : Winlogon Cached Password Weakness\", \"severity\": 0, \"vpr_score\": null, \"plugin_id\": 11457, \"severity_index\": 4, \"cpe\": null, \"offline\": false, \"plugin_family\": \"Windows\", \"snoozed\": 0}]}",
    "@timestamp" => 2023-04-25T15:05:59.802Z,
          "path" => "/testELK/exemple.json"
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 25, 2023, 4:16pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/35 "2023-04-25T16:16:10Z")

</div>

That [[main]\<file] error you were getting before, it is coming from the codec in the file input. The  
[[main]\>worker0] error that was coming from the json filter no longer occurs. That's all expected. What do you see in elasticsearch?

It just occurred to me what the problem might be. When logstash first sees the file it starts reading at the beginning. It remembers how far it has gotten in the sincedb. This is persisted across restarts. `start_position => "beginning"` only has any effect the first time the file input sees the file.

Suppose you have a file

```
{ "foo": "bar" }

```

and logstash reads all 15 bytes. If you edit the file to contain

```
{ "foo": "bar", "hello", "world" }

```

then logstash will read `"hello", "world" }` and the json code will produce an error very similar to the one you are seeing.

Try adding `sincedb_path => "/dev/null"` to your file input and see if a pipeline restart correctly decodes the JSON.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 26, 2023, 7:54am UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/36 "2023-04-26T07:54:21Z")

</div>

In elastic I saw this in mapping

```auto
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "message": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "path": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "tags": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [April 26, 2023, 8:25am UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/37 "2023-04-26T08:25:37Z")

</div>

Hi, thank you for unblocking me. I have a question: when my index is created in Elasticsearch, all the data is put together. For example, for the index field, all the indexes are grouped together when they should be separated. We should have multiple lines of data in Elasticsearch, but instead we have only one. And in the fields, each field contains all the data. Here's an example of the index field so you can see what I mean.  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/1/113c338dbf17d9247e63b29953732bffcc75958d.png)

I only have on line of data :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f14293e55cac06a5b861289b5f60b71158bac0ae.png)

How can I fix this, please?

---

<div class="post-metadata">

**Author:** ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\
**Post date:** [April 26, 2023, 10:35am UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/38 "2023-04-26T10:35:58Z")

</div>

Do you mean that they are different sets of data, or is it all expressed in a single JSON object? Because Elasticsearch stores the data in JSON format, so for the field

```auto
vulnerabilities.vuln_[414,396,....]

```

what it does is store everything as an array of data from the same field, if I'm not mistaken it does so because it considers that it is the same field only that it contains N different values, hence you see a field with N values in it.

---

<div class="post-metadata">

**Author:** ![Hanni](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@Hanni](https://discuss.elastic.co/u/Hanni)\
**Post date:** [May 2, 2023, 1:42pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/39 "2023-05-02T13:42:19Z")

</div>

hello, I solved my problem. Thank you for your help. But I have a question how can I do if I want to send many json file in Elasticsearch?

---

<div class="post-metadata">

**Author:** ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\
**Post date:** [May 4, 2023, 12:13pm UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/40 "2023-05-04T12:13:18Z")

</div>

Hi, all the json files are located in the same folder?

If they are in the same folder you mai config the input to read all .json files like this

```auto
path => "/testELK/*.json"

```

---

<div class="post-metadata">

**Author:** ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\
**Post date:** [May 10, 2023, 6:34am UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/41 "2023-05-10T06:34:23Z")

</div>

It solves your problem or still having?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 7, 2023, 6:35am UTC](https://discuss.elastic.co/t/issue-with-logsatsh/330227/42 "2023-06-07T06:35:16Z")

</div>

Closed on June 7, 2023, 6:35am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/issue-with-logsatsh/330227.md?page=1)