Grok filter to Replace @timestamp with my log-time

Toni_Riera · October 26, 2017, 7:49am

Hi there,

I'm trying to find out what to do, but got stucked in there. I've got ELK runnning on a single machine, and i'm trying to analyze certain logs using Logstash.

Currently i've got an entry like this:

Jan 22 15:42:40 raspberrypi sshd[954]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key

So i'm trying to filter the log-date and replace @timestamp with it.

My logstash-test.conf looks like this:

input {
file {
path => "/var/log/banana/auth.log" #This is the log trying to parse
}

}

filter {

    grok {
            match => { "message" => "%{MONTHNUM2:monthnum} %{MONTHDAY:monthday} %{TIME:time} %{GREEDYDATA:syslog_message}" } #Try to identify the data to the replace @timestamp

    }


    mutate {
            add_field => { "newtimestamp" => "%{monthnum} %{monthday} %{time}" } #Adding a new variable
    }

    date {
            match => [ "newtimestamp" , "ISO8601" , "MMM dd HH:mm:ss" ]
            target => "@timestamp"  #Replacing timestamp
            locale => "es"
            timezone => "UTC"
    }

}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}

So, the thing is that i keep gettint the same value of @timestamp (The time the entry is given to ELK, not the Log data), is it there anyway i can do it?

Thanks.

magnusbaeck · October 26, 2017, 8:35am

Please show an example entry that you think is incorrect. Copy/paste from Kibana's JSON tab.

Toni_Riera · October 26, 2017, 9:01am

There you have:

{
"_index": "logstash-2017.10.24",
"_type": "logs",
"_id": "AV9NsSkvfU4Yos9JWWi7",
"_version": 1,
"_score": null,
"_source": {
"path": "/var/log/banana/auth.log",
"@timestamp": "2017-10-24T09:22:56.889Z",
"@version": "1",
"host": "elk",
"message": "Sep 22 21:44:33 raspberrypi sudo: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/bin/bash",
"type": "logs",
"tags": [
"_grokparsefailure",
"_geoip_lookup_failure"
]
},
"fields": {
"@timestamp": [
1508836976889
]
},
"sort": [
1508836976889
]
}

magnusbaeck · October 26, 2017, 10:04am

The date filter is failing because the grok filter is failing and doesn't extract the timestamp into its own field. The documentation contains an example of how to deal with syslog messages: https://www.elastic.co/guide/en/logstash/current/config-examples.html

Toni_Riera · November 7, 2017, 12:06pm

Thanks for the reply,

My previous problem wasn't all about the .conf file, ELK wasn't reading it. Now it is, but i simplified the .conf file, in order to add a new field with the new time i'd like to check, but on logs keeps apearing the same error.

Here you have the .con file:

input {
file {
path => "/home/soft/logs/banana/auth.log"
}
}

filter {

    grok {
            match => { "message" => " %{MONTH:month}%{MONTHDAY:day}%{TIME:time}}
    }   

    mutate {
            add_field =>  { "newtimestamp" => "%{month}%{day}%{time}"}
    }

}
output {
elasticsearch {
hosts => ["localhost:9200"]
}

    stdout { codec => rubydebug }

}

And the logs Error:
[2017-11-07T13:03:52,405][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, {, } at line 14, column 20 (byte 190) after filter {\n\n\tgrok {\n\t\tmatch => { "message" => " %{MONTH:month}%{MONTHDAY:day}%{TIME:time} }\n\t}\n\n\tmutate {\n\t\tadd_field => { ""}

Toni_Riera · November 7, 2017, 12:17pm

I was missing a " on the line that the log's said the .conf file was wrong, but after it, restarting elasticsearch, kibana and logstash, the index doesn't appear to be existing at all. I'm looking for the 'logstash-*' index.

Have you got any idea why doesn't exists?

Thanks,

magnusbaeck · November 8, 2017, 6:20am

Elasticsearch doesn't delete indexes on its own, so either you deleted it or the account you're using to access ES doesn't have read permissions to the index(es).

Toni_Riera · November 13, 2017, 3:08pm

Actually i'm reading the logs to see what's really happening on it. And i find that logstashs really reads the file.

After all the "Adding pattern" lines, one it starts with the replacement_pattern says that Grok Compiled OK, and starts the pipeline, and it keeps going like that all the way up:

2017-11-13T15:58:17,376][DEBUG][logstash.filters.grok ?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]u [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok [2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok [2017-11-13T15:58:17,378][DEBUG][logstash.filters.grok nth>\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|rua o?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b) 01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[ [2017-11-13T15:58:17,382][INFO ][logstash.pipeline =>5, "pipeline.max_inflight"=>250}
[2017-11-13T15:58:18,666][INFO ][logstash.pipeline [2017-11-13T15:58:18,699][DEBUG][logstash.inputs.file [2017-11-13T15:58:18,699][DEBUG][logstash.inputs.file [2017-11-13T15:58:18,709][DEBUG][logstash.inputs.file [2017-11-13T15:58:18,710][DEBUG][logstash.inputs.file [2017-11-13T15:58:18,710][DEBUG][logstash.inputs.file [2017-11-13T15:58:18,722][DEBUG][logstash.agent [2017-11-13T15:58:18,726][DEBUG][logstash.agent [2017-11-13T15:58:18,727][DEBUG][logstash.api.service [2017-11-13T15:58:18,844][INFO ][logstash.agent [2017-11-13T15:58:23,696][DEBUG][logstash.pipeline [2017-11-13T15:58:28,699][DEBUG][logstash.pipeline [2017-11-13T15:58:32,739][DEBUG][logstash.inputs.file [2017-11-13T15:58:33,699][DEBUG][logstash.pipeline [2017-11-13T15:58:38,700][DEBUG][logstash.pipeline [2017-11-13T15:58:43,699][DEBUG][logstash.pipeline [2017-11-13T15:58:47,752][DEBUG][logstash.inputs.file [2017-11-13T15:58:48,699][DEBUG][logstash.pipeline [2017-11-13T15:58:53,699][DEBUG][logstash.pipeline [2017-11-13T15:58:58,699][DEBUG][logstash.pipeline [2017-11-13T15:59:02,765][DEBUG][logstash.inputs.file [2017-11-13T15:59:03,699][DEBUG][logstash.pipeline [2017-11-13T15:59:08,700][DEBUG][logstash.pipeline [2017-11-13T15:59:13,701][DEBUG][logstash.pipeline [2017-11-13T15:59:17,797][DEBUG][logstash.inputs.file [2017-11-13T15:59:18,702][DEBUG][logstash.pipeline [2017-11-13T15:59:23,702][DEBUG][logstash.pipeline [2017-11-13T15:59:28,704][DEBUG][logstash.pipeline [2017-11-13T15:59:32,808][DEBUG][logstash.inputs.file [2017-11-13T15:59:33,704][DEBUG][logstash.pipeline [2017-11-13T15:59:38,705][DEBUG][logstash.pipeline [2017-11-13T15:59:43,706][DEBUG][logstash.pipeline e ] replacement_pattern => (?MONTH:month\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|Mm?r(?:ch|z)
l(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|Oo?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b)
] replacement_pattern => (?MONTHDAY:day(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))
] replacement_pattern => (?TIME:time(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))
] replacement_pattern => (?:(?:2[0123]|[01]?[0-9]))
] replacement_pattern => (?:(?:[0-5][0-9]))
] replacement_pattern => (?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))
] Grok compiled OK {:pattern=>"%{MONTH:month} %{MONTHDAY:day} %{TIME:time}", :expanded_pattern=>"(?<MONTH:mo
r)?|Mm?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|O
(?MONTHDAY:day(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?TIME:time(?!<[0-9])(?:(?:2[0123]|[
0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9]))"}
] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"
] Pipeline main started
] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
] _discover_file: /var/log/banana/auth.log: new: /var/log/banana/auth.log (exclude is [])
] _open_file: /var/log/banana/auth.log: opening
] /var/log/banana/auth.log: sincedb last value 1262243, cur size 1262243
] /var/log/banana/auth.log: sincedb: seeking to 1262243
] Starting puma
] Trying to start WebServer {:port=>9600}
] [api-service] start
] Successfully started Logstash API endpoint {:port=>9600}
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
] Pushing flush onto pipeline
] Pushing flush onto pipeline
] Pushing flush onto pipeline

I think i can't create the index on Kibana because no files or data are indexed on it, so the index doesn't really exists, but i don't understand what's going on with reading the file and trying to parse it. Is it really parsing the file?

Thanks Magnus,

magnusbaeck · November 13, 2017, 5:11pm

So is anything being added to /var/log/banana/auth.log? Logstash is tailing the file. With your configuration it won't read it from the top.

Toni_Riera · November 14, 2017, 11:23am

But on my .conf file, after the input path, i've got the line start from the beggining, so should be reading the hole file from the top.

Isn't it?

magnusbaeck · November 14, 2017, 12:01pm

You mean you have start_position => beginning? Not according to what you posted earlier which is the only information I have.

start_position => beginning only matters for new files that Logstash hasn't seen before. In this case Logstash has already recorded the end of the file as the current position so introducing start_position => beginning won't make a difference. You need to clear the sincedb entry or stop using sincedb to clear this. See the file input documentation for details or check out the archives. It has been covered here countless times before.

Toni_Riera · November 14, 2017, 12:34pm

That really helped me, thank you so much.

One more last question about the .conf file.

As i said before, i'd like to create a new field with the logs time, my .con looks like that now:

input {
file {
path => "/var/log/banana/auth.log"
start_position => "beginning"
}
}

filter {

    grok {
            match => { "message" => "%{MONTH:month} %{MONTHDAY:day} %{TIME:time}"}
    }

    mutate {
            add_field =>  { "newtimestamp" => "%{MONTH} %{MONTHNUM} %{TIME}"}
    }

}
output {
elasticsearch {
hosts => ["localhost:9200"]
}

    stdout { codec => rubydebug }

}

Once i add the new field, wich is created it's content it's literally they parsing 'code', i mean here you have the JSON output, no errors, it's about quotes i think, is it?

{
"_index": "logstash-2017.11.14",
"_type": "logs",
"_id": "AV-6gGWS1TaAY5OfLJU6",
"_version": 1,
"_score": null,
"_source": {
"path": "/var/log/banana/auth.log",
"@timestamp": "2017-11-14T12:28:17.502Z",
"month": "Sep",
"@version": "1",
"host": "0.0.0.0",
"newtimestamp": "%{MONTH} %{MONTHNUM} %{TIME}",
"time": "21:30:46",
"message": "Sep 22 21:30:46 raspberrypi systemd-logind[565]: New session c2 of user pi.",
"day": "22"
},
"fields": {
"@timestamp": [
1510662497502
]
},
"sort": [
1510662497502
]
}

new timestamp was suposed to be Sep 22 21:30:46.

Thank you so much.

magnusbaeck · November 14, 2017, 2:05pm

        add_field =>  { "newtimestamp" => "%{MONTH} %{MONTHNUM} %{TIME}"}

You're conflating the grok pattern names with the field names. This needs to be %{month} %{day} %{time}. However, you're inventing the wheel; how to parse syslog files is one of the examples in the Logstash documentation (https://www.elastic.co/guide/en/logstash/current/config-examples.html).

system · December 12, 2017, 2:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.