Logstash Permission Issue

Elastic Stack Logstash
1

Error Message

[2023-11-22T13:15:51,460][WARN ][filewatch.sincedbcollection][main][fd97ffae0e8f2b3b8d71c9b308ee7a3feac45d9133bd3968160c580a4d2e603e] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,553][WARN ][filewatch.sincedbcollection][main][ad3090afce627ecb703f9c0c122b9730e4f70cc54b5c8aea4292d762ed2cb7b5] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,467][WARN ][filewatch.sincedbcollection][main][a6054bf1bfecbdbaf2036fe164abf8820b5e5b66eaea9cd9bf70b5df7e0fdef9] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,467][WARN ][filewatch.sincedbcollection][main][cca0a03eb06c263369f45f79d110821f474f346c38245e1faff1509025eda359] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,466][WARN ][filewatch.sincedbcollection][main][f2b53b9270e894436c1334cc8f3ad4512b4ae8f94da62b8c3e295ddc4a270120] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,465][WARN ][filewatch.sincedbcollection][main][45bcb6b54d20132262b921d180f4e6f7cb460f5aff5359930f8b67a02bda8a0f] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb_error_nginx error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,465][WARN ][filewatch.sincedbcollection][main][ffe46720a323d0bae9738865bffac9fc23a265d924290f9054c3cca9cf16f6b1] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,465][WARN ][filewatch.sincedbcollection][main][249eaff860b8d192975ac60f154b65d125021d827971ed93d4c7f3fc9a93ce92] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,464][WARN ][filewatch.sincedbcollection][main][a4ff897cc5b6cc9c286a41fb326c0558fcbe6bb8b87f82d3a116f5642de3cded] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,463][WARN ][filewatch.sincedbcollection][main][c0c41963f527909742ff23a859fa86c036a87bbf274606e9c57c2527cc6f953c] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,460][WARN ][filewatch.sincedbcollection][main][97d1fb9c415f4fcf93eab1e105e8a572db3f4de92da33d5f0a57a7d247d563b0] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"}
[2023-11-22T13:15:51,657][WARN ][filewatch.sincedbcollection][main][f975a8dd4a6660695774459d7b45792e78dd2413254c2a4444a0cf22bcdd4292] sincedb_write: unable to write atomically due to permissions error, falling back to non-atomic write: /common/logs/.sincedb_json error: {:exception=>Errno::EPERM, :message=>"Operation not permitted - No message available"

Logstash configuration

   file {
        path => "/common/logs/service/app.log"
        start_position => "beginning"
        sincedb_path => "/common/logs/.sincedb"
        type => "service"
        codec => multiline {
          pattern => "^%{TIMESTAMP_ISO8601}"
          negate => true
          what => previous
          max_lines => 1000
        }
      }

Permission for sincedb inside the container.

bash-4.4$ ls -ltra /common/logs
total 20
drwxrwxrwx  2 logstash logstash   32 Nov 16 00:37 kafka
drwxrwxrwx  2 logstash logstash    6 Nov 16 01:57 netapp-service
drwxrwxrwx  2      421 root       28 Nov 16 01:58 oracle- trigger
drwxrwxrwx  3 root     root       55 Nov 22 12:54 ..
-rwxr-xr-x  1 logstash logstash    0 Nov 22 16:33 .sincedb_error_nginx
-rwxr-xr-x  1 logstash logstash  517 Nov 22 16:33 .sincedb_json
-rwxr-xr-x  1 logstash logstash   76 Nov 22 16:33 .sincedb
drwxrwxrwx 13 root     logstash  273 Nov 22 16:33 .

bash-4.4$ ls -ld  /common
drwxrwxrwx 3 root root 55 Nov 22 12:54 /shared
2

Is /common a network drive?

3

Yes it is a remote drive.

4

The file input does not work well on network drives and can have some issues, probably this is one of them.

Try to change the path of the since db file to a local path, like /var/lib/logstash/

5

oh I understand. But why we planned to have it this way because this setup is on container and we need sincedb in the remote disc, if the containers terminates and new container comes up it will look into remote path. if the path is local then it will create a problem.

6

@leandrojmp
Could you please help me with any other alternative.
having sincedb path specifying in any other config file will help ?

Issue with logstash on aarch64 - #15 by doctorhey

7

I'm not sure this will work, as mentioned the file input does not work well with network paths, it has a couple of issues as mentioned in the documentation and probably many others since it is not qwll tested in network paths.

Unfortunately I do not use containers, so I do not have much experience with running Logstash on containers.

One thing that you could do is to consume the logs with a normal VM and send those logs to Kafka Topics, then use Logstash on containers to consume from those Kafka Topics, apply your filters and send to your outputs.

8

@leandrojmp Thank you for the suggestion.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.