Ingest node / source name

Elastic Stack Elasticsearch
1

Hello, i am using filebeat for sending logs to elastic and ingest node for parsing with grok.
my logs have 2 fields for datestamp - > 1/2 is in the name of every file (Year-month-day-hour) and
1/2 is in the beginning of every string (minute-second).
Trying to figure how to extract this properly.
This is how its now:

"grok" : {
"field" : "source",
"patterns" : [
"%{INT:_ingest.tempyymmddhh}.log"
]
}
},
{
"set" : {
"field" : "_ingest.tempdate",
"value" : "{{_ingest.tempyymmddhh}}{{_ingest.tempmm}}{{_ingest.tempss}}"
}
},
{
"date" : {
"field" : "_ingest.tempdate",
"target_field" : "@timestamp",
"formats" : [
"yyMMddHHmmss.SSSSSS"
],
but i am getting error: "field [source] not present as part of path [source]"
For testing puposes i can set source offset like this
"source": "/path/*/smth.log"

But how should i set it for live version with working filebeat and new log files every hour. Any tips?

2

can you provide a sample document and the ingest pipeline so others can reproduce the issue?

Also the elasticsearch version you are using would be beneficial to know.

Thanks!

3

i am using elastic 7.2.0

Unfortunetly I cant provide sample and honestly I dont have problems with parsing(so i supose there is no need in full file. the only problem is how to extract datestamp from filename + string.
For example i have filename: 19073122.log (which is year/month/day/hour)
and string in file name that starts with: 00:13.375000 (which is minute/second/mlsec)

So i know i can put source in offset section. But how should i do that if source changes a lot and it comes from filebeat (7.3.0)

4

hmm.. i cant copy full ingest node. its more than 7k characters. and looks like you can upload only image.

5

i suppose i can PM you my full node if u ok with it

6

I'm still not sure I got all the information I need. do you send the information of year/month/day AND minute/second/msec as part of the document or not?

7

I send year/month/day/hour as name of the document
and min/sec/msec as start of almost every string in the document

8

so accessing the _id plus a field name would be sufficient. Do you happen to have a sample document at hand with a sample id and a sample field that could be used?

9

idk abut _id field. But i think i found an answer!
instead of "source" field i started using "log.path" according to exported-fields
and now i am parsing filename correctly.

Sorry for inconvenience i caused you. i am just starting to use elastic and not fully understand some things and jargon

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.