Cannot search /var/log/messages without '*' wildcard before string

Elastic Stack Beats
1

Version 6.2.2

System module enabled. When searching in Discover /var/log/messages returns zero results. However, the syslog messages are showing in Discover. If I search for user I get results. If searching for user* there are no results, as an example.

2

Can you post some screenshots to demonstrate the issue? It might be that the field you are trying to search by is not searchable.

3

Example syslog message:

system.syslog.message (?@123.456.789.000) [WARNING] Authentication failed for user

The message only appeared when I searched for user

4

Does source:"/var/log/messages" work in Kibana?

5

When pasting source:"/var/log/messages" into the search field messages do show. They also show naturally with no search string. When searching with a search string the source of /var/log/messages does not return results. Hopefully my explanation is clear.

6

The source field is a keyword, which means it's not searched by default when no field is specified. Isn't searching with source:"/var/log/messages" an acceptable solution?

If not, I think a solution would be to modify the Elasticsearch mapping template and add an equivalent text field by using copy to.

The approximate process for that would be:

GET /_template/filebeat-6.2.2

<edit the template file>

PUT /_template/filebeat-6.2.2
{
 <modified template>
}

Note that the template applies only index creation, so the modified template will be applied only when a new index is template.

Also, note that we encode the version in the template name, which means you'll have to do this on each upgrade.

We do hope to make this process easier with a feature in Beats (basically specified which fields should be copied to a text field), but it's not currently available.

7

Let me explain the use case - We pull logs from many log file sources via the filebeat.yml and simply want to type the text word we are after. Example: We want to find all instances of the text "user" in all log files. If I do that now with using the system module all the log files will be searched except syslog since it is using the system module? Do I understand this correctly?

Would it just be simpler to then use the filebeat.yml to point to /var/log/messages instead of using the module?

8

Ah, I realised we have a bug in the module. The message field would make more sense as a text here: https://github.com/elastic/beats/blob/master/filebeat/module/system/syslog/_meta/fields.yml#L18-L20

I'll open a PR to fix it.

In the mean time, you can use a normal input/prospector instead of the module to workaround, you just lose the parsed program name, pid, etc.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.