Hi guys,
I am new to ELK and I need your help in order to parse a JSON file with Logstash.
I want to make it available in Kibana.
My logstash.conf is
input {
file {
path => "/usr/share/logstash/c.json"
start_position => beginning
mode => "read"
codec => "json"
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash_internal"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
}
}
Ths is the config from docker-compose.yml
logstash:
build:
context: logstash/
args:
ELASTIC_VERSION: ${ELASTIC_VERSION}
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
- ./logstash/pipeline:/usr/share/logstash/pipeline
- ./logstash/c.json:/usr/share/logstash/c.json
ports:
- "5044:5044"
- "5000:5000/tcp"
- "5000:5000/udp"
- "9600:9600"
environment:
LS_JAVA_OPTS: -Xmx256m -Xms256m
LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
networks:
- elk
depends_on:
- elasticsearch
This is the error I receive:
[logstash.codecs.json ]
JSON parse error, original data now in message field
{:message=>"incompatible json object type=java.lang.String ,
only hash map or arrays are supported", :exception=>LogStash::Json::ParserError,
:data=>" \"site\": \"mitre\""}
This is the way how my json looks like:
{
"data": {
"webs": {
"data": [
{
"id": "75735",
"attributes": {
"cveID": "CVE-2022-24724",
"feed": "Nist",
"dateAdded": "2022-04-02",
"severity": 9.8,
"shortDescription": " cmark-gfm is GitHub s extended version of the C reference implementation of CommonMark. Prior to versions 0.29.0.gfm.3 and 0.28.3.gfm.21 an integer overflow in cmark-gfm s table row parsing table.c row from string may lead to heap memory corruption when parsing tables who s marker rows contain more than UINT16 MAX columns. The impact of this heap corruption ranges from Information Leak to Arbitrary Code Execution depending on how and where cmark-gfm is used. If cmark-gfm is used for rendering remote user controlled markdown this vulnerability may lead to Remote Code Execution RCE in applications employing affected versions of the cmark-gfm library. This vulnerability has been patched in the following cmark-gfm versions 0.29.0.gfm.3 and 0.28.3.gfm.21. A workaround is available. The vulnerability exists in the table markdown extensions of cmark-gfm. Disabling the table extension will prevent this vulnerability from being triggered. ",
"product": "NETWORK",
"vendorProject": " github ",
"projectName": " cmark-gfm ",
"source": "https://nvd.nist.gov/vuln/detail/CVE-2022-24724",
"dbID": 136756,
"vulnerabilityName": null,
"requiredAction": null,
"dueDate": null,
"site": "nist"
}
},
{
"id": "75734",
"attributes": {
"cveID": "CVE-2022-21698",
"feed": "Nist",
"dateAdded": "2022-04-02",
"severity": 7.5,
"shortDescription": " client golang is the instrumentation library for Go applications in Prometheus and the promhttp package in client golang provides tooling around HTTP servers and clients. In client golang prior to version 1.11.1 HTTP server is susceptible to a Denial of Service through unbounded cardinality and potential memory exhaustion when handling requests with non-standard HTTP methods. In order to be affected an instrumented software must use any of promhttp.InstrumentHandler middleware except RequestsInFlight not filter any specific methods e.g GET before middleware pass metric with method label name to our middleware and not have any firewall LB proxy that filters away requests with unknown method . client golang version 1.11.1 contains a patch for this issue. Several workarounds are available including removing the method label name from counter gauge used in the InstrumentHandler turning off affected promhttp handlers adding custom middleware before promhttp handler that will sanitize the request method given by Go http.Request and using a reverse proxy or web application firewall configured to only allow a limited set of methods. ",
"product": "NETWORK",
"vendorProject": " prometheus ",
"projectName": " client golang ",
"source": "https://nvd.nist.gov/vuln/detail/CVE-2022-21698",
"dbID": 134416,
"vulnerabilityName": null,
"requiredAction": null,
"dueDate": null,
"site": "nist"
}
},
}
It is like the tool doesn't know how to do a map/loop over data.webs.data and instead is doing map over all array elements and not objects as it is intended to do.
My Dockerfile
ARG ELASTIC_VERSION
# https://www.docker.elastic.co/
FROM docker.elastic.co/logstash/logstash:${ELASTIC_VERSION}
# Add your logstash plugins setup here
RUN logstash-plugin install logstash-codec-json
I can see some data in Kibana, but is not well formated, is like one document with { in it, another one with a different field, but is not saved as real objects, but as part of the object.
If you can help, I will owe you a lot!
Later Edit: I've changed the json file to ndjson and not it takes all the data as one single collection 
I need to make the tool to split them.
{"webs":{"data":[{"id":"75735","attributes":{"cveID":"CVE-2022-24724","feed":"Nist","dateAdded":"2022-04-02","severity":9.8,"shortDescription":" cmark-gfm is GitHub s extended version of the C reference implementation of CommonMark. Prior to versions 0.29.0.gfm.3 and 0.28.3.gfm.21 an integer overflow in cmark-gfm s table row parsing table.c row from string may lead to heap memory corruption when parsing tables who s marker rows contain more than UINT16 MAX columns. The impact of this heap corruption ranges from Information Leak to Arbitrary Code Execution depending on how and where cmark-gfm is used. If cmark-gfm is used for rendering remote user controlled markdown this vulnerability may lead to Remote Code Execution RCE in applications employing affected versions of the cmark-gfm library. This vulnerability has been patched in the following cmark-gfm versions 0.29.0.gfm.3 and 0.28.3.gfm.21. A workaround is available. The vulnerability exists in the table markdown extensions of cmark-gfm. Disabling the table extension will prevent this vulnerability from being triggered. ","product":"NETWORK","vendorProject":" github ","projectName":" cmark-gfm ","source":"https://nvd.nist.gov/vuln/detail/CVE-2022-24724","dbID":136756,"vulnerabilityName":null,"requiredAction":null,"dueDate":null,"site":"nist"}},{"id":"75734","attributes":{"cveID":"CVE-2022-21698","feed":"Nist","dateAdded":"2022-04-02","severity":7.5,"shortDescription":" client golang is the instrumentation library for Go applications in Prometheus and the promhttp package in client golang provides tooling around HTTP servers and clients. In client golang prior to version 1.11.1 HTTP server is susceptible to a Denial of Service through unbounded cardinality and potential memory exhaustion when handling requests with non-standard HTTP methods. In order to be affected an instrumented software must use any of promhttp.InstrumentHandler middleware except RequestsInFlight not filter any specific methods e.g GET before middleware pass metric with method label name to our middleware and not have any firewall LB proxy that filters away requests with unknown method . client golang version 1.11.1 contains a patch for this issue. Several workarounds are available including removing the method label name from counter gauge used in the InstrumentHandler turning off affected promhttp handlers adding custom middleware before promhttp handler that will sanitize the request method given by Go http.Request and using a reverse proxy or web application firewall configured to only allow a limited set of methods. ","product":"NETWORK","vendorProject":" prometheus ","projectName":" client golang ","source":"https://nvd.nist.gov/vuln/detail/CVE-2022-21698","dbID":134416,"vulnerabilityName":null,"requiredAction":null,"dueDate":null,"site":"nist"}},{"id":"76647","attributes":{"cveID":"CVE-2022-1201 ","feed":"Mitre","dateAdded":"2022-04-02","severity":null,"shortDescription":"CVE-2022-1201 NULL Pointer Dereference in mrb_vm_exec with super in GitHub repository mruby/mruby prior to 3.2. This vulnerability is capable of making the mruby interpreter crash, thus affecting the availability of the system. https://t.co/XDHtH1N8R1","product":null,"vendorProject":null,"projectName":null,"source":"https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1201","dbID":null,"vulnerabilityName":null,"requiredAction":null,"dueDate":null,"site":"mitre"}}]}}