Hello, I am currently working on a subject. I am trying to parse my data in JSON format to store it in Elasticsearch, but Logstash is unable to parse my data and is generating errors. Can you help me?
Are you using json filter?
filter {
json {
source => "message"
}
}
Can you share the filter settings and an example of data?
1 Like
No I am using the codec => json
here is my conf :
input {
file {
path => "/testELK/file.json"
start_position => "beginning"
type => "json"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "nindex"
}
stdout {codec => rubydebug}
}
try setting after input and before output config filter like this
input{
....
}
filter {
json {
source => "message"
}
}
output{
....
}
if the input data is a json object that only comes as plain text with this filter plugin it can be transformed into a json object in elasticsearch
Should I remove the codec json?
When I launch Logstash, it gets stuck at this point and I don't understand why
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2023-04-19 12:14:56.415 [main] runner - Starting Logstash {"logstash.version"=>"7.17.9", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.18+10 on 11.0.18+10 +indy +jit [linux-x86_64]"}
[INFO ] 2023-04-19 12:14:56.420 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2023-04-19 12:14:56.658 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-04-19 12:14:57.716 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601, :ssl_enabled=>false}
[INFO ] 2023-04-19 12:14:58.719 [Converge PipelineAction::Create] Reflections - Reflections took 82 ms to scan 1 urls, producing 119 keys and 419 values
[WARN ] 2023-04-19 12:14:59.386 [Converge PipelineAction::Create] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-04-19 12:14:59.421 [Converge PipelineAction::Create] file - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-04-19 12:14:59.478 [Converge PipelineAction::Create] plain - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-04-19 12:14:59.517 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-04-19 12:14:59.706 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["http://localhost:9200"]}
[INFO ] 2023-04-19 12:14:59.993 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[WARN ] 2023-04-19 12:15:00.276 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2023-04-19 12:15:00.288 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.2.2) {:es_version=>8}
[WARN ] 2023-04-19 12:15:00.290 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2023-04-19 12:15:00.327 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. data_stream => auto resolved to false
[WARN ] 2023-04-19 12:15:00.331 [[main]-pipeline-manager] json - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-04-19 12:15:00.381 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[INFO ] 2023-04-19 12:15:00.420 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/fich.conf"], :thread=>"#<Thread:0x54214d3b run>"}
[INFO ] 2023-04-19 12:15:01.091 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.67}
[INFO ] 2023-04-19 12:15:01.134 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_25912788a7501d0a199f1d6fef237025", :path=>["/testELK/NMS_Basic_Network_Scan_ARCADI_37432.json"]}
[INFO ] 2023-04-19 12:15:01.150 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-04-19 12:15:01.228 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[INFO ] 2023-04-19 12:15:01.239 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
Here is my json data
{
"creation_date": "2023-01-04T15:30:28.000000Z",
"prioritization": {
"plugins": [
{
"hosts": [
{
"host_ip": "",
"host_fqdn": null,
"hostname": "",
"id": 1
},
{
"host_ip": "",
"host_fqdn": null,
"hostname": "",
"id": 2
},
{
"host_ip": "",
"host_fqdn": null,
"hostname": "",
"id": 4
},
{
"host_ip": "",
"host_fqdn": null,
"hostname": "",
"id": 7
}
],
"severity": 4,
"pluginname": "windows",
"pluginattributes": {
"threat_intensity_last_28": "Very Low",
"synopsis": "The remote Windows host is affected by multiple vulnerabilities.",
"description": "The remote Windows host is missing security update 5022289. It is, therefore, affected by multiple vulnerabilities\n\n - Microsoft ODBC Driver Remote Code Execution Vulnerability (CVE-2023-21732)\n\n - Microsoft WDAC OLE DB provider for SQL Server Remote Code Execution Vulnerability (CVE-2023-21681)\n\n - Windows Advanced Local Procedure Call (ALPC) Elevation of Privilege Vulnerability (CVE-2023-21674)\n\nNote that Nessus has not tested for these issues but has instead relied only on the application's self-reported version number.",
"risk_information": {
"cvss_temporal_vector": "CVSS2#E:H/RL:OF/RC:C",
"stig_severity": "I",
"risk_factor": "Critical",
"cvss_vector": "CVSS2#AV:N/AC:L/Au:N/C:C/I:C/A:C",
"cvss3_base_score": "9.1",
"cvss_temporal_score": "8.7",
"cvss_base_score": "10.0",
"cvss3_temporal_vector": "CVSS:3.0/E:H/RL:O/RC:C",
"cvss3_temporal_score": "8.7",
"cvss3_vector": "CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:H"
},
"ref_information": {
"ref": [
{
"name": "msft",
"values": {
"value": [
"MS23-5022289"
]
},
"url": "http://***"
},
{
"name": "cisa-known-exploited",
"values": {
"value": [
"2023/01/31"
]
}
},
{
"name": "iava",
"values": {
"value": [
"2023-A-0025",
"2023-A-0027"
]
}
},
{
"name": "mskb",
"values": {
"value": [
"5022289",
"5022289"
]
},
"url": "https://support.microsoft.com/en-us/help/"
},
{
"name": "cve",
"values": {
"value": [
"CVE-2023-21776",
"CVE-2023-21774",
"CVE-2023-21773",
"CVE-2023-21772",
"CVE-2023-21767",
"CVE-2023-21766",
"CVE-2023-21765",
"CVE-2023-21760",
"CVE-2023-21758",
"CVE-2023-21757",
"CVE-2023-21754",
"CVE-2023-21752",
"CVE-2023-21750",
"CVE-2023-21749",
"CVE-2023-21748",
"CVE-2023-21747",
"CVE-2023-21746",
"CVE-2023-21739",
"CVE-2023-21732",
"CVE-2023-21730",
"CVE-2023-21728",
"CVE-2023-21726",
"CVE-2023-21683",
"CVE-2023-21682",
"CVE-2023-21681",
"CVE-2023-21680",
"CVE-2023-21679",
"CVE-2023-21678",
"CVE-2023-21677",
"CVE-2023-21675",
"CVE-2023-21674",
"CVE-2023-21563",
"CVE-2023-21561",
"CVE-2023-21560",
"CVE-2023-21558",
"CVE-2023-21557",
"CVE-2023-21556",
"CVE-2023-21555",
"CVE-2023-21552",
"CVE-2023-21549",
"CVE-2023-21548",
"CVE-2023-21547",
"CVE-2023-21546",
"CVE-2023-21543",
"CVE-2023-21542",
"CVE-2023-21541",
"CVE-2023-21537",
"CVE-2023-21535",
"CVE-2023-21532",
"CVE-2023-21527",
"CVE-2023-21525",
"CVE-2023-21524"
]
},
"url": "http://web.nvd.nist.gov/view/vuln/detail?vulnId="
}
]
},
"threat_sources_last_28": "No recorded events",
"plugin_name": "KB5022289: Windows 10 Version 1607 and Windows Server 2016 Security Update (January 2023)",
"vpr_score": "9.2",
"cvss_score_source": "CVE-2023-21732",
"see_also": [
"January 10, 2023—KB5022289 (OS Build 14393.5648) - Microsoft Support"
],
"product_coverage": "Very High",
"iava": "2023-A-0027",
"cisa-known-exploited": "2023/01/31",
"threat_recency": "30 to 120 days",
"offline": true,
"cvss3_score_source": "CVE-2023-21557",
"fname": "smb_nt_ms23_jan_5022289.nasl",
"cvssV3_impactScore": "6",
"plugin_information": {
"plugin_version": "1.5",
"plugin_id": 169779,
"plugin_type": "local",
"plugin_publication_date": "2023/01/10",
"plugin_family": "Windows : Microsoft Bulletins",
"plugin_modification_date": "2023/01/18"
},
"msft": "MS23-5022289",
"solution": "Apply Security Update 5022289",
"vuln_information": {
"exploitability_ease": "Exploits are available",
"cpe": "cpe:/o:microsoft:windows",
"exploit_available": "true",
"vuln_publication_date": "2023/01/10",
"patch_publication_date": "2023/01/10"
},
"age_of_vuln": "60 - 180 days",
"exploit_code_maturity": "Functional"
},
"pluginfamily": "Windows : Microsoft Bulletins",
"host_count": 4,
"pluginid": "169779"
}
],
"threat_level": 4
},
"vulnerabilities": [
{
"count": 1,
"score": null,
"vuln_index": 414,
"plugin_name": "WordPad History",
"severity": 0,
"vpr_score": null,
"plugin_id": 92438,
"severity_index": 0,
"cpe": "cpe:/o:microsoft:windows",
"offline": false,
"plugin_family": "Windows",
"snoozed": 0
},
{
"count": 1,
"score": null,
"vuln_index": 396,
"plugin_name": "Windows Defender Installed",
"severity": 0,
"vpr_score": null,
"plugin_id": 131023,
"severity_index": 1,
"cpe": "cpe:/a:microsoft:windows_defender",
"offline": false,
"plugin_family": "Windows",
"snoozed": 0
}
]
}
If ends with this
[INFO ] 2023-04-19 12:15:01.228 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=> ""}
it starts succesfully
try with this configuration
input {
file {
codec => "json"
path => "/testELK/file.json"
start_position => "beginning"
#type => "json"
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "nindex"
...
}
stdout {
codec => rubydebug
}
}
Complete the output and change the source if need
I had tried it but the mapping is not correctly done according to my data. Here is the mapping of my index in elasticsearch :
{
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Plugin ID": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"nom": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
I can't see the fields of my JSON in the mapping
I have try this in one cluster and its fails if contains this format of dates
"vuln_publication_date": "2023-01-10",
"patch_publication_date": "2023-01-10"
If contains this dates its prop a failure like this
..... ent_exception", "reason"=>"mapper [prioritization.plugins.pluginattributes.ref_information.ref.values.value] cannot be changed from type [text] to [date]"}}}
the configuration to map all fields is
input {
...
}
filter {
json {
source => "message"
}
}
output {
...
}
if you need the date you must parse the date first remove from message and then use json filter plugin.
the data without date in the format "yyyy-mm-dd" after json filter plugin stored into elasticsearch
"prioritization": {
"threat_level": 4,
"plugins": [
{
"severity": 4,
"pluginname": "windows",
"hosts": [
{
"host_ip": "",
"hostname": "",
"host_fqdn": null,
"id": 1
},
{
"host_ip": "",
"hostname": "",
"host_fqdn": null,
"id": 2
},
{
"host_ip": "",
"hostname": "",
"host_fqdn": null,
"id": 4
},
{
"host_ip": "",
"hostname": "",
"host_fqdn": null,
"id": 7
}
],
"pluginid": "169779",
"host_count": 4,
"pluginfamily": "Windows:MicrosoftBulletins",
"pluginattributes": {
"fname": "smb_nt_ms23_jan_5022289.nasl",
"cvss3_score_source": "CVE-2023-21557",
"risk_information": {
"cvss3_base_score": "9.1",
"cvss_base_score": "10.0",
"cvss_vector": "CVSS2#AV:N/AC:L/Au:N/C:C/I:C/A:C",
"risk_factor": "Critical",
"stig_severity": "I",
"cvss_temporal_score": "8.7",
"cvss3_temporal_score": "8.7",
"cvss3_vector": "CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:H",
"cvss3_temporal_vector": "CVSS:3.0/E:H/RL:O/RC:C",
"cvss_temporal_vector": "CVSS2#E:H/RL:OF/RC:C"
},
"threat_intensity_last_28": "VeryLow",
"msft": "MS23-5022289",
"description": "TheremoteWindowshostismissingsecurityupdate5022289.Itis,therefore,affectedbymultiplevulnerabilities\n\n-MicrosoftODBCDriverRemoteCodeExecutionVulnerability(CVE-2023-21732)\n\n-MicrosoftWDACOLEDBproviderforSQLServerRemoteCodeExecutionVulnerability(CVE-2023-21681)\n\n-WindowsAdvancedLocalProcedureCall(ALPC)ElevationofPrivilegeVulnerability(CVE-2023-21674)\n\nNotethatNessushasnottestedfortheseissuesbuthasinsteadreliedonlyontheapplication'sself-reportedversionnumber.",
"threat_sources_last_28": "Norecordedevents",
"vuln_information": {
"exploit_available": "true",
"exploitability_ease": "Exploitsareavailable",
"cpe": "cpe:/o:microsoft:windows"
},
"synopsis": "TheremoteWindowshostisaffectedbymultiplevulnerabilities.",
"plugin_name": "KB5022289:Windows10Version1607andWindowsServer2016SecurityUpdate(January2023)",
"product_coverage": "VeryHigh",
"ref_information": {
"ref": [
{
"values": {
"value": [
"MS23-5022289"
]
},
"name": "msft",
"url": "http://***"
},
{
"values": {
"value": [
"2023-A-0025",
"2023-A-0027"
]
},
"name": "iava"
},
{
"values": {
"value": [
"5022289",
"5022289"
]
},
"name": "mskb",
"url": "https://support.microsoft.com/en-us/help/"
},
{
"values": {
"value": [
"CVE-2023-21776",
"CVE-2023-21774",
"CVE-2023-21773",
"CVE-2023-21772",
"CVE-2023-21767",
"CVE-2023-21766",
"CVE-2023-21765",
"CVE-2023-21760",
"CVE-2023-21758",
"CVE-2023-21757",
"CVE-2023-21754",
"CVE-2023-21752",
"CVE-2023-21750",
"CVE-2023-21749",
"CVE-2023-21748",
"CVE-2023-21747",
"CVE-2023-21746",
"CVE-2023-21739",
"CVE-2023-21732",
"CVE-2023-21730",
"CVE-2023-21728",
"CVE-2023-21726",
"CVE-2023-21683",
"CVE-2023-21682",
"CVE-2023-21681",
"CVE-2023-21680",
"CVE-2023-21679",
"CVE-2023-21678",
"CVE-2023-21677",
"CVE-2023-21675",
"CVE-2023-21674",
"CVE-2023-21563",
"CVE-2023-21561",
"CVE-2023-21560",
"CVE-2023-21558",
"CVE-2023-21557",
"CVE-2023-21556",
"CVE-2023-21555",
"CVE-2023-21552",
"CVE-2023-21549",
"CVE-2023-21548",
"CVE-2023-21547",
"CVE-2023-21546",
"CVE-2023-21543",
"CVE-2023-21542",
"CVE-2023-21541",
"CVE-2023-21537",
"CVE-2023-21535",
"CVE-2023-21532",
"CVE-2023-21527",
"CVE-2023-21525",
"CVE-2023-21524"
]
},
"name": "cve",
"url": "http://web.nvd.nist.gov/view/vuln/detail?vulnId="
}
]
},
"see_also": [
"January10,2023—KB5022289(OSBuild14393.5648)-MicrosoftSupport"
],
"offline": true,
"cvss_score_source": "CVE-2023-21732",
"iava": "2023-A-0027",
"threat_recency": "30to120days",
"cisa-known-exploited": "2023-01-31",
"exploit_code_maturity": "Functional",
"solution": "ApplySecurityUpdate5022289",
"plugin_information": {
"plugin_id": 169779,
"plugin_family": "Windows:MicrosoftBulletins",
"plugin_version": "1.5",
"plugin_type": "local"
},
"cvssV3_impactScore": "6",
"age_of_vuln": "60-180days",
"vpr_score": "9.2"
}
}
]
}
and this is de mapping
{
"mappings": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"norms": false,
"type": "text"
}
}
},
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"norms": false,
"type": "text"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"agent": {
"properties": {
"ephemeral_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"creation_date": {
"type": "date"
},
"ecs": {
"properties": {
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"event": {
"properties": {
"original": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"fields": {
"properties": {
"aplicacion": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"geoip": {
"dynamic": "true",
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
}
}
},
"host": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"input": {
"properties": {
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"log": {
"properties": {
"file": {
"properties": {
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"offset": {
"type": "long"
}
}
},
"message": {
"type": "text",
"norms": false
},
"prioritization": {
"properties": {
"plugins": {
"properties": {
"host_count": {
"type": "long"
},
"hosts": {
"properties": {
"host_ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"hostname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"id": {
"type": "long"
}
}
},
"pluginattributes": {
"properties": {
"age_of_vuln": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cisa-known-exploited": {
"type": "date"
},
"cvss3_score_source": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvssV3_impactScore": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss_score_source": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"description": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"exploit_code_maturity": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"fname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"iava": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"msft": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"offline": {
"type": "boolean"
},
"plugin_information": {
"properties": {
"plugin_family": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"plugin_id": {
"type": "long"
},
"plugin_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"plugin_version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"plugin_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"product_coverage": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"ref_information": {
"properties": {
"ref": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"url": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"values": {
"properties": {
"value": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
}
}
}
}
},
"risk_information": {
"properties": {
"cvss3_base_score": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss3_temporal_score": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss3_temporal_vector": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss3_vector": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss_base_score": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss_temporal_score": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss_temporal_vector": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"cvss_vector": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"risk_factor": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"stig_severity": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
},
"see_also": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"solution": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"synopsis": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"threat_intensity_last_28": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"threat_recency": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"threat_sources_last_28": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"vpr_score": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"vuln_information": {
"properties": {
"cpe": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"exploit_available": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"exploitability_ease": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
}
}
}
}
},
"pluginfamily": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"pluginid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"pluginname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"severity": {
"type": "long"
}
}
},
"threat_level": {
"type": "long"
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"vulnerabilities": {
"properties": {
"count": {
"type": "long"
},
"cpe": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"offline": {
"type": "boolean"
},
"plugin_family": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"plugin_id": {
"type": "long"
},
"plugin_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"norms": false
},
"severity": {
"type": "long"
},
"severity_index": {
"type": "long"
},
"snoozed": {
"type": "long"
},
"vuln_index": {
"type": "long"
}
}
}
}
}
}
If I understood you correctly, the issue is with the date format, right?
I tested with your JSON file but I still have the same problem. My Logstash is not mapping. Is there a configuration to add in Logstash configurations to enable it to process JSON files? I installed my Logstash via apt
The data that i use is the next
{"creation_date": "2023-01-04T15:30:28","prioritization": {"plugins": [{"hosts": [{"host_ip": "","host_fqdn": null,"hostname": "","id": 1},{"host_ip": "","host_fqdn": null,"hostname": "","id": 2},{"host_ip": "","host_fqdn": null,"hostname": "","id": 4},{"host_ip": "","host_fqdn": null,"hostname": "","id": 7}],"severity": 4,"pluginname": "windows","pluginattributes": {"threat_intensity_last_28": "VeryLow","synopsis": "TheremoteWindowshostisaffectedbymultiplevulnerabilities.","description": "TheremoteWindowshostismissingsecurityupdate5022289.Itis,therefore,affectedbymultiplevulnerabilities\n\n-MicrosoftODBCDriverRemoteCodeExecutionVulnerability(CVE-2023-21732)\n\n-MicrosoftWDACOLEDBproviderforSQLServerRemoteCodeExecutionVulnerability(CVE-2023-21681)\n\n-WindowsAdvancedLocalProcedureCall(ALPC)ElevationofPrivilegeVulnerability(CVE-2023-21674)\n\nNotethatNessushasnottestedfortheseissuesbuthasinsteadreliedonlyontheapplication'sself-reportedversionnumber.","risk_information": {"cvss_temporal_vector": "CVSS2#E:H/RL:OF/RC:C","stig_severity": "I","risk_factor": "Critical","cvss_vector": "CVSS2#AV:N/AC:L/Au:N/C:C/I:C/A:C","cvss3_base_score": "9.1","cvss_temporal_score": "8.7","cvss_base_score": "10.0","cvss3_temporal_vector": "CVSS:3.0/E:H/RL:O/RC:C","cvss3_temporal_score": "8.7","cvss3_vector": "CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:H"},"ref_information": {"ref": [{"name": "msft","values": {"value": ["MS23-5022289"]},"url": "http://***"},{"name": "iava","values": {"value": ["2023-A-0025","2023-A-0027"]}},{"name": "mskb","values": {"value": ["5022289","5022289"]},"url": "https://support.microsoft.com/en-us/help/"},{"name": "cve","values": {"value": ["CVE-2023-21776","CVE-2023-21774","CVE-2023-21773","CVE-2023-21772","CVE-2023-21767","CVE-2023-21766","CVE-2023-21765","CVE-2023-21760","CVE-2023-21758","CVE-2023-21757","CVE-2023-21754","CVE-2023-21752","CVE-2023-21750","CVE-2023-21749","CVE-2023-21748","CVE-2023-21747","CVE-2023-21746","CVE-2023-21739","CVE-2023-21732","CVE-2023-21730","CVE-2023-21728","CVE-2023-21726","CVE-2023-21683","CVE-2023-21682","CVE-2023-21681","CVE-2023-21680","CVE-2023-21679","CVE-2023-21678","CVE-2023-21677","CVE-2023-21675","CVE-2023-21674","CVE-2023-21563","CVE-2023-21561","CVE-2023-21560","CVE-2023-21558","CVE-2023-21557","CVE-2023-21556","CVE-2023-21555","CVE-2023-21552","CVE-2023-21549","CVE-2023-21548","CVE-2023-21547","CVE-2023-21546","CVE-2023-21543","CVE-2023-21542","CVE-2023-21541","CVE-2023-21537","CVE-2023-21535","CVE-2023-21532","CVE-2023-21527","CVE-2023-21525","CVE-2023-21524"]},"url": "http://web.nvd.nist.gov/view/vuln/detail?vulnId="}]},"threat_sources_last_28": "Norecordedevents","plugin_name": "KB5022289:Windows10Version1607andWindowsServer2016SecurityUpdate(January2023)","vpr_score": "9.2","cvss_score_source": "CVE-2023-21732","see_also": ["January10,2023—KB5022289(OSBuild14393.5648)-MicrosoftSupport"],"product_coverage": "VeryHigh","iava": "2023-A-0027","cisa-known-exploited": "2023-01-31","threat_recency": "30to120days","offline": true,"cvss3_score_source": "CVE-2023-21557","fname": "smb_nt_ms23_jan_5022289.nasl","cvssV3_impactScore": "6","plugin_information": {"plugin_version": "1.5","plugin_id": 169779,"plugin_type": "local","plugin_family": "Windows:MicrosoftBulletins"},"msft": "MS23-5022289","solution": "ApplySecurityUpdate5022289","vuln_information": {"exploitability_ease": "Exploitsareavailable","cpe": "cpe:/o:microsoft:windows","exploit_available": "true"},"age_of_vuln": "60-180days","exploit_code_maturity": "Functional"},"pluginfamily": "Windows:MicrosoftBulletins","host_count": 4,"pluginid": "169779"}],"threat_level": 4},"vulnerabilities": [{"count": 1,"score": null,"vuln_index": 414,"plugin_name": "WordPadHistory","severity": 0,"vpr_score": null,"plugin_id": 92438,"severity_index": 0,"cpe": "cpe:/o:microsoft:windows","offline": false,"plugin_family": "Windows","snoozed": 0},{"count": 1,"score": null,"vuln_index": 396,"plugin_name": "WindowsDefenderInstalled","severity": 0,"vpr_score": null,"plugin_id": 131023,"severity_index": 1,"cpe": "cpe:/a:microsoft:windows_defender","offline": false,"plugin_family": "Windows","snoozed": 0}]}
to use de json plugin you dont need to add any configuration
how can I do with my date
Maybe you must to use the mutate plugin and change the format of the dates or split into pieces and then change the format of the date or extract into other variable after join all the pieces whitout the dates and after apli the json filter plugin.
Try it, I'll try when I have time to find a solution.
hello still have the same problem with my json
Hi, try to map the variables to string all of them that contains the date in the format "yyyy-mm-dd" in elasticsearch before runing Logstash.
I've seem that the problem is elasticsearch that in the variables who has not mapping try to map into date and the format is not valid.
Do you know the date format supported by Elasticsearch?