Timestamp in json logs

Elastic Stack Beats
1

Hi,
I am using Ossec https://www.ossec.net/docs/ to log alerts locally to alerts.json which already has a timestamp in the log. Can I use this timestamp in the logstash output to elastic ?

The input is below. All my beats aggregate to logstash then split or direct events to SUMO, Elastic or whatever is needed.

{"timestamp":"2018-07-31T15:56:06.373-0400","rule":{"level":3,"description":"Windows: Service startup type was changed." ...., "blaa"}

filebeat.inputs:

  • type: log
    json.keys_under_root: true
    json.add_error_key: true
    json.overwrite_keys: true
    enabled: true
    paths:
    • /var/ossec/logs/alerts/alerts.json
2

How is the alerts.json file being created in this scenario. By default Logstash will use the @timestamp field as the main timestamp associated with a given Event.

3

The file is generated directly from ossec/bin/ossec-analysisd daemon on a linux host.
What I am seeing in Kibana and when I dump to file from Logstash are two timestamps.
The second coming from logstash.

4

{
"_index": "filebeat-6.3.2-2018.07.31",
"_type": "doc",
"_id": "6n0U8mQBl-NcLeOlLsVz",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-07-31T20:52:29.452Z",
"tags": [
"MOC",
"OSSEC",
"beats_input_raw_event"
],
"predecoder": {
"program_name": "WinEvtLog",
"timestamp": "2018 Jul 31 16:52:11",
"hostname": "##################"
},
"offset": 57557220,
"timestamp": "2018-07-31T16:52:29.62-0400",
"beat": {
"version": "6.3.2",
"hostname": "###############",
"name": "################"
},
"location": "WinEvtLog",
"id": "1533070349.403263",
"full_log": "2018 Jul 31 16:52:11 WinEvtLog: System: INFORMATION(7040): Service Control Manager: SYSTEM: NT AUTHORITY: #####################: The start type of the Background Intelligent Transfer Service service was changed from auto start to demand start. ",
"input": {
"type": "log"
},
"prospector": {
"type": "log"
},
"manager": {
"name": "################"
},
"data": {
"dstuser": "SYSTEM",
"data": "Service Control Manager",
"id": "7040",
"type": "System",
"system_name": "########################",
"status": "INFORMATION"
},
"source": "/var/ossec/logs/alerts/alerts.json",
"@version": "1",
"host": {
"name": "############"
},
"decoder": {
"parent": "windows",
"name": "windows"
},
"agent": {
"id": "114"
},
"rule": {
"groups": [
"windows",
"policy_changed",
"hipaa_164.308.a.3.ii.B",
"hipaa_164.308.a.4.i",
"hipaa_164.308.a.4.ii.B",
"hipaa_164.308.a.4.ii.C",
"hipaa_164.312.a.2.i",
"hipaa_164.312.a.2.iv",
"nist_800_53_ac-02",
"hipaa_164.308.a.5.ii.C",
"hipaa_164.312.b",
"nist_800_53_au-06",
"hipaa_164.308.a.1.ii.D",
"hipaa_164.312.b",
"nist_800_53_au-07",
"hipaa_164.308.a.5.ii.B",
"nist_800_53_si-04"
],
"pci_dss": [
"10.6"
],
"gdpr": [
"2.3"
],
"mail": false,
"id": "18145",
"level": 3,
"firedtimes": 699,
"description": "Windows: Service startup type was changed.",
"info": "This does not appear to be logged on Windows 2000."
}
},
"fields": {
"@timestamp": [
"2018-07-31T20:52:29.452Z"
]
},
"sort": [
1533070349452
]
}

5

I found this in another thread. Do I need json.overwitekeys in my beats-pipeline.conf ?

6

You just need the processor section. You can set overwrite keys in that if you need it, but you shouldn't set it in the top level. See the Decode JSON fields docs.

7

Ok that may be my problem, it's in the input section and have no processor section.

filebeat.inputs:
- type: log
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true
  enabled: true
  paths:
    - /var/ossec/logs/alerts/alerts.json
#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["###.###.###.###:5044"]
8

Followed this thread

9

Little confused

json.overwrite_keys in filebeat.reference.yml has it in the input section as well as much of the discussions I have looked at.

Release notes have it in the processor section but there is no context of the difference.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.