# Filebeat resent all events of a logfile - every day **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 24, 2020, 8:06am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478 "2020-07-24T08:06:38Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![scheffler](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@scheffler](https://discuss.elastic.co/u/scheffler)\ **Post date:** [July 24, 2020, 8:06am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/1 "2020-07-24T08:06:38Z")
Hi The logfile has always the same name, but a housekeeping job copy and zip the file every night, with the side effect that the file gets a new inode. I guess that the inode is harvesters primary key in the registry and filebeat assumes a new file - don't care about the same name. tail\_files: true in filebeat.yml doesn't help, because it is a new file for filebeat. A test for this effect is when you change a logfile for test cases with vi. If you save the file with :wq it gets a new i node. (stat before and start after saving). Elastic shows all entries of the file, but if you use the pipe echo 'errormessage' \>\> filename, only the last error will send. Has anybody an idea to solve the problem? Cheers, Heinz ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [July 27, 2020, 8:38am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/2 "2020-07-27T08:38:13Z")
Take a look at `file_identity` property: [https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html) ---
**Author:** ![scheffler](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@scheffler](https://discuss.elastic.co/u/scheffler)\ **Post date:** [July 28, 2020, 6:12am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/3 "2020-07-28T06:12:03Z")
Ok, this seems to be the solution (from the discryption point of view). I put much time in it - with no solution. For my case I have exact path/filenames without wildcards, so file\_identity.path should be the way. I tried every possible configuration regarding the file\_identity.path property, but it doesn't work. One new entry with vi and filebeat sents all again. There is no documentation about the value of path. I found just one example: file\_identity.path: ~ I tried file\_identity.path: file\_identity.path: ~ file\_identity.path: true file\_identity.path: 'path/filename' but... nothing works for me. What means ~ ? ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [July 28, 2020, 8:01am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/4 "2020-07-28T08:01:42Z")
Can you post the whole config you're using? The property `file_identity.path: ~` should be fine. ---
**Author:** ![scheffler](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@scheffler](https://discuss.elastic.co/u/scheffler)\ **Post date:** [July 28, 2020, 1:39pm UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/5 "2020-07-28T13:39:20Z")
This is the relevant section: ```auto filebeat.inputs: - type: log enabled: true paths: - /disk00/app/oracle/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log - /disk00/app/oracle/diag/rdbms/aitdev01/AITDEV01/trace/alert_AITDEV01.log - /disk00/app/oracle/diag/rdbms/aitdev02/AITDEV02/trace/alert_AITDEV02.log - /disk00/app/oracle/diag/rdbms/aitma02/AITMA02/trace/alert_AITMA02.log - /disk00/app/oracle/diag/rdbms/aitprd01/AITPRD01/trace/alert_AITPRD01.log - /disk00/app/oracle/diag/rdbms/aitprd02/AITPRD02/trace/alert_AITPRD02.log - /disk00/app/oracle/diag/rdbms/cusprd01/CUSPRD01/trace/alert_CUSPRD01.log - /disk00/app/oracle/diag/rdbms/cusprd03/CUSPRD03/trace/alert_CUSPRD03.log - /disk00/app/oracle/diag/rdbms/cusprd05/CUSPRD05/trace/alert_CUSPRD05.log - /disk00/app/oracle/diag/rdbms/cusqa02/CUSQA02/trace/alert_CUSQA02.log - /disk00/app/oracle/diag/rdbms/gfadev/GFADEV/trace/alert_GFADEV.log - /disk00/app/oracle/diag/rdbms/gfaprd/GFAPRD/trace/alert_GFAPRD.log - /disk00/app/oracle/diag/rdbms/hipadb/HIPADB/trace/alert_HIPADB.log - /disk00/app/oracle/diag/rdbms/prodb/PRODB/trace/alert_PRODB.log - /home/delphe/dummy.log #tail_files: true file_identity.path: ~ include_lines: ['ORA-'] multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}' # Regex for the pattern which matches the beginning of the interesting logentry multiline.negate: true multiline.match: after # Example of an error log entry: # 2020-04-19T06:00:00.497793+02:00 # Errors in file /disk00/app/oracle/diag/rdbms/aitdev01/AITDEV01/trace/AITDEV01_j003_143635.trc: # ORA-12012: error on auto execute of job "SYS"."ORA$AT_SQ_SQL_SW_4628" # ORA-38153: Software edition is incompatible with SQL plan management. # ORA-06512: at "SYS.DBMS_SPM_INTERNAL", line 6202 # ORA-06512: at "SYS.DBMS_SPM", line 2806 # ORA-06512: at line 34 # 2020-04-19T06:00:19.115017+02:00 # Timestamp i.e. begin of the next error. The last entry will not appear since the logentry is the last one. processors: - copy_fields: fields: - from: message to: oracle.message fail_on_error: false ignore_missing: true - truncate_fields: fields: - message max_characters: 33 fail_on_error: false ignore_missing: true - timestamp: field: message layouts: - '2020-04-19T06:00:00.497793+02:00' target_field: oracle.timestamp # Maybe not the best solution, but current state of my knowledge. More add_fields processor insctances to write the ORACLE-Instance in a new field called "oracle.instance.name" - add_fields: when: contains: log.file.path: "ASM1" target: oracle.instance fields: name: "ASM1" - add_fields: when: contains: log.file.path: "AITDEV01" target: oracle.instance fields: name: "AITDEV01" - add_fields: when: contains: log.file.path: "AITDEV02" target: oracle.instance fields: name: "AITDEV02" - add_fields: when: contains: log.file.path: "AITMA02" target: oracle.instance fields: name: "AITMA02" - add_fields: when: contains: log.file.path: "AITPRD01" target: oracle.instance fields: name: "AITPRD01" - add_fields: when: contains: log.file.path: "AITPRD02" target: oracle.instance fields: name: "AITPRD02" - add_fields: when: contains: log.file.path: "CUSPRD01" target: oracle.instance fields: name: "CUSPRD01" - add_fields: when: contains: log.file.path: "CUSPRD05" target: oracle.instance fields: name: "CUSPRD05" - add_fields: when: contains: log.file.path: "CUSPRD03" target: oracle.instance fields: name: "CUSPRD03" - add_fields: when: contains: log.file.path: "CUSQA02" target: oracle.instance fields: name: "CUSQA02" - add_fields: when: contains: log.file.path: "GFAPRD" target: oracle.instance fields: name: "GFAPRD" - add_fields: when: contains: log.file.path: "PRODB" target: oracle.instance fields: name: "PRODB" - add_fields: when: contains: log.file.path: "HIPADB" target: oracle.instance fields: name: "HIPADB" - add_fields: when: contains: log.file.path: "GFADEV" target: oracle.instance fields: name: "GFADEV" # dummy scans the file /home/delphe/dummy.log for test cases - add_fields: when: contains: log.file.path: "dummy" target: oracle.instance fields: name: "dummy" #============================= Filebeat modules =============================== filebeat.config.modules: .... ``` Thanks, Heinz ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [July 28, 2020, 3:18pm UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/6 "2020-07-28T15:18:26Z")
One more thing. Which version of filebeat are you using? As the `file_identity` is really fresh and I believe it's currently available on master and 7.x branched. It didn't become a part of any release yet. ---
**Author:** ![scheffler](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@scheffler](https://discuss.elastic.co/u/scheffler)\ **Post date:** [July 29, 2020, 5:40am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/7 "2020-07-29T05:40:17Z")
Hi Marcin We use 7.6.2 Cheers, Heinz ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [July 29, 2020, 6:58am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/8 "2020-07-29T06:58:01Z")
You can try to build the latest master and verify if that version works for you. ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [July 29, 2020, 12:04pm UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/9 "2020-07-29T12:04:39Z")
Have you tried to exclude the copied files from harvesting with `exclude_files` option? ---
**Author:** ![scheffler](https://avatars.discourse-cdn.com/v4/letter/s/bb73d2/32.png) [@scheffler](https://discuss.elastic.co/u/scheffler)\ **Post date:** [July 30, 2020, 8:57am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/10 "2020-07-30T08:57:43Z")
Thanks for the hint, but I use path/filenames without wildcards and the ziped files are in a complete different directory. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 27, 2020, 10:57am UTC](https://discuss.elastic.co/t/filebeat-resent-all-events-of-a-logfile-every-day/242478/11 "2020-08-27T10:57:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.