Grok filter to Replace @timestamp with my log-time

Elastic Stack Logstash
1

Hi there,

I'm trying to find out what to do, but got stucked in there. I've got ELK runnning on a single machine, and i'm trying to analyze certain logs using Logstash.

Currently i've got an entry like this:

Jan 22 15:42:40 raspberrypi sshd[954]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key

So i'm trying to filter the log-date and replace @timestamp with it.

My logstash-test.conf looks like this:

input {
file {
path => "/var/log/banana/auth.log" #This is the log trying to parse
}

}

filter {

    grok {
            match => { "message" => "%{MONTHNUM2:monthnum} %{MONTHDAY:monthday} %{TIME:time} %{GREEDYDATA:syslog_message}" } #Try to identify the data to the replace @timestamp

    }


    mutate {
            add_field => { "newtimestamp" => "%{monthnum} %{monthday} %{time}" } #Adding a new variable
    }

    date {
            match => [ "newtimestamp" , "ISO8601" , "MMM dd HH:mm:ss" ]
            target => "@timestamp"  #Replacing timestamp
            locale => "es"
            timezone => "UTC"
    }

}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}

So, the thing is that i keep gettint the same value of @timestamp (The time the entry is given to ELK, not the Log data), is it there anyway i can do it?

Thanks.

2

Please show an example entry that you think is incorrect. Copy/paste from Kibana's JSON tab.

3

There you have:

{
"_index": "logstash-2017.10.24",
"_type": "logs",
"_id": "AV9NsSkvfU4Yos9JWWi7",
"_version": 1,
"_score": null,
"_source": {
"path": "/var/log/banana/auth.log",
"@timestamp": "2017-10-24T09:22:56.889Z",
"@version": "1",
"host": "elk",
"message": "Sep 22 21:44:33 raspberrypi sudo: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/bin/bash",
"type": "logs",
"tags": [
"_grokparsefailure",
"_geoip_lookup_failure"
]
},
"fields": {
"@timestamp": [
1508836976889
]
},
"sort": [
1508836976889
]
}

4

The date filter is failing because the grok filter is failing and doesn't extract the timestamp into its own field. The documentation contains an example of how to deal with syslog messages: https://www.elastic.co/guide/en/logstash/current/config-examples.html

5

Thanks for the reply,

My previous problem wasn't all about the .conf file, ELK wasn't reading it. Now it is, but i simplified the .conf file, in order to add a new field with the new time i'd like to check, but on logs keeps apearing the same error.

Here you have the .con file:

input {
file {
path => "/home/soft/logs/banana/auth.log"
}
}

filter {

    grok {
            match => { "message" => " %{MONTH:month}%{MONTHDAY:day}%{TIME:time}}
    }   

    mutate {
            add_field =>  { "newtimestamp" => "%{month}%{day}%{time}"}
    }

}
output {
elasticsearch {
hosts => ["localhost:9200"]
}

    stdout { codec => rubydebug }

}

And the logs Error:
[2017-11-07T13:03:52,405][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, {, } at line 14, column 20 (byte 190) after filter {\n\n\tgrok {\n\t\tmatch => { "message" => " %{MONTH:month}%{MONTHDAY:day}%{TIME:time} }\n\t}\n\n\tmutate {\n\t\tadd_field => { ""}

7

I was missing a " on the line that the log's said the .conf file was wrong, but after it, restarting elasticsearch, kibana and logstash, the index doesn't appear to be existing at all. I'm looking for the 'logstash-*' index.

Have you got any idea why doesn't exists?

Thanks,

8

Elasticsearch doesn't delete indexes on its own, so either you deleted it or the account you're using to access ES doesn't have read permissions to the index(es).

9

Actually i'm reading the logs to see what's really happening on it. And i find that logstashs really reads the file.

After all the "Adding pattern" lines, one it starts with the replacement_pattern says that Grok Compiled OK, and starts the pipeline, and it keeps going like that all the way up:

2017-11-13T15:58:17,376][DEBUG][logstash.filters.grok ] replacement_pattern => (?MONTH:month\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|Mm?r(?:ch|z)
?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|Oo?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b)
[2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok ] replacement_pattern => (?MONTHDAY:day(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))
[2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok ] replacement_pattern => (?TIME:time(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))
[2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok ] replacement_pattern => (?:(?:2[0123]|[01]?[0-9]))
[2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok ] replacement_pattern => (?:(?:[0-5][0-9]))
[2017-11-13T15:58:17,377][DEBUG][logstash.filters.grok ] replacement_pattern => (?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))
[2017-11-13T15:58:17,378][DEBUG][logstash.filters.grok ] Grok compiled OK {:pattern=>"%{MONTH:month} %{MONTHDAY:day} %{TIME:time}", :expanded_pattern=>"(?<MONTH:mo
nth>\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|Mm?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|O
o?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b) (?MONTHDAY:day(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?TIME:time(?!<[0-9])(?:(?:2[0123]|[
01]?[0-9])):(?:(?:[0-5][0-9]))(?::(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))(?![0-9]))"}
[2017-11-13T15:58:17,382][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"
=>5, "pipeline.max_inflight"=>250}
[2017-11-13T15:58:18,666][INFO ][logstash.pipeline ] Pipeline main started
[2017-11-13T15:58:18,699][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
[2017-11-13T15:58:18,699][DEBUG][logstash.inputs.file ] _discover_file: /var/log/banana/auth.log: new: /var/log/banana/auth.log (exclude is [])
[2017-11-13T15:58:18,709][DEBUG][logstash.inputs.file ] _open_file: /var/log/banana/auth.log: opening
[2017-11-13T15:58:18,710][DEBUG][logstash.inputs.file ] /var/log/banana/auth.log: sincedb last value 1262243, cur size 1262243
[2017-11-13T15:58:18,710][DEBUG][logstash.inputs.file ] /var/log/banana/auth.log: sincedb: seeking to 1262243
[2017-11-13T15:58:18,722][DEBUG][logstash.agent ] Starting puma
[2017-11-13T15:58:18,726][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2017-11-13T15:58:18,727][DEBUG][logstash.api.service ] [api-service] start
[2017-11-13T15:58:18,844][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-11-13T15:58:23,696][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:28,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:32,739][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
[2017-11-13T15:58:33,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:38,700][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:43,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:47,752][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
[2017-11-13T15:58:48,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:53,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:58:58,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:02,765][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
[2017-11-13T15:59:03,699][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:08,700][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:13,701][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:17,797][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
[2017-11-13T15:59:18,702][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:23,702][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:28,704][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:32,808][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/banana/auth.log: glob is: ["/var/log/banana/auth.log"]
[2017-11-13T15:59:33,704][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:38,705][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-11-13T15:59:43,706][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
e

I think i can't create the index on Kibana because no files or data are indexed on it, so the index doesn't really exists, but i don't understand what's going on with reading the file and trying to parse it. Is it really parsing the file?

Thanks Magnus,

10

So is anything being added to /var/log/banana/auth.log? Logstash is tailing the file. With your configuration it won't read it from the top.

11

But on my .conf file, after the input path, i've got the line start from the beggining, so should be reading the hole file from the top.

Isn't it?

12

You mean you have start_position => beginning? Not according to what you posted earlier which is the only information I have.

start_position => beginning only matters for new files that Logstash hasn't seen before. In this case Logstash has already recorded the end of the file as the current position so introducing start_position => beginning won't make a difference. You need to clear the sincedb entry or stop using sincedb to clear this. See the file input documentation for details or check out the archives. It has been covered here countless times before.

1 Like
13

That really helped me, thank you so much.

One more last question about the .conf file.

As i said before, i'd like to create a new field with the logs time, my .con looks like that now:

input {
file {
path => "/var/log/banana/auth.log"
start_position => "beginning"
}
}

filter {

    grok {
            match => { "message" => "%{MONTH:month} %{MONTHDAY:day} %{TIME:time}"}
    }

    mutate {
            add_field =>  { "newtimestamp" => "%{MONTH} %{MONTHNUM} %{TIME}"}
    }

}
output {
elasticsearch {
hosts => ["localhost:9200"]
}

    stdout { codec => rubydebug }

}

Once i add the new field, wich is created it's content it's literally they parsing 'code', i mean here you have the JSON output, no errors, it's about quotes i think, is it?

{
"_index": "logstash-2017.11.14",
"_type": "logs",
"_id": "AV-6gGWS1TaAY5OfLJU6",
"_version": 1,
"_score": null,
"_source": {
"path": "/var/log/banana/auth.log",
"@timestamp": "2017-11-14T12:28:17.502Z",
"month": "Sep",
"@version": "1",
"host": "0.0.0.0",
"newtimestamp": "%{MONTH} %{MONTHNUM} %{TIME}",
"time": "21:30:46",
"message": "Sep 22 21:30:46 raspberrypi systemd-logind[565]: New session c2 of user pi.",
"day": "22"
},
"fields": {
"@timestamp": [
1510662497502
]
},
"sort": [
1510662497502
]
}

new timestamp was suposed to be Sep 22 21:30:46.

Thank you so much.

14 
        add_field =>  { "newtimestamp" => "%{MONTH} %{MONTHNUM} %{TIME}"}

You're conflating the grok pattern names with the field names. This needs to be %{month} %{day} %{time}. However, you're inventing the wheel; how to parse syslog files is one of the examples in the Logstash documentation (https://www.elastic.co/guide/en/logstash/current/config-examples.html).

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.