Logstash 7.9.1 have issues in ARM

Elastic Stack Logstash
1 
[2020-09-23T00:58:12,480][ERROR][logstash.javapipeline ][main][167820c884ed4020909a7d2491232c40eac5b47a3f91495a85db21bdf02afeca] A plugin had an unrecoverable error. Will restart this plugin.

Pipeline_id:main

Plugin: <LogStash::Inputs::File start_position=>"end", path=>["/applications.logs/*"], codec=><LogStash::Codecs::Multiline pattern=>"^%{MONTHDAY} %{MONTH} %{YEAR} %{TIME}", what=>"previous", id=>"186ada5d-8731-4a2a-97c2-9fbd48d1b5e1", negate=>true, enable_metric=>true, charset=>"UTF-8", multiline_tag=>"multiline", max_lines=>500, max_bytes=>10485760>, exclude=>["*.gz"], id=>"167820c884ed4020909a7d2491232c40eac5b47a3f91495a85db21bdf02afeca", sincedb_path=>"tmp/mysincedbfile", enable_metric=>true, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false, check_archive_validity=>false>

Error: Operation not permitted - No message available

Exception: Errno::EPERM

Stack: org/jruby/RubyFile.java:675:in `chown'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/helper.rb:41:in `write_atomically'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/sincedb_collection.rb:232:in `atomic_write'

org/jruby/RubyMethod.java:131:in `call'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/sincedb_collection.rb:216:in `sincedb_write'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/sincedb_collection.rb:190:in `flush_at_interval'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/sincedb_collection.rb:32:in `request_disk_flush'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/handlers/base.rb:76:in `controlled_read'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/handlers/grow.rb:10:in `block in handle_specifically'

org/jruby/RubyKernel.java:1442:in `loop'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/handlers/grow.rb:7:in `handle_specifically'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/handlers/base.rb:25:in `handle'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/processor.rb:43:in `grow'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/processor.rb:234:in `block in process_active'

org/jruby/RubyArray.java:1809:in `each'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/processor.rb:228:in `process_active'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/tail_mode/processor.rb:75:in `process_all_states'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/watch.rb:67:in `iterate_on_state'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/watch.rb:44:in `subscribe'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/filewatch/observing_tail.rb:12:in `subscribe'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.1/lib/logstash/inputs/file.rb:364:in `run'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/logstash-core/lib/logstash/java_pipeline.rb:378:in `inputworker'

/aarch64/Logstash-agent/Logstash-agent-6624.0-0/logstash/logstash-core/lib/logstash/java_pipeline.rb:369:in `block in start_input'

[2020-09-23T00:58:13,509][INFO ][filewatch.observingtail ][main][167820c884ed4020909a7d2491232c40eac5b47a3f91495a85db21bdf02afeca] QUIT - closing all files and shutting down.

[2020-09-23T00:58:13,513][INFO ][filewatch.observingtail ][main][167820c884ed4020909a7d2491232c40eac5b47a3f91495a85db21bdf02afeca] START, creating Discoverer, Watch with file and sincedb collections

Can anyone help in resolving this error?

2

Did you mean that to be /tmp?

sincedb_path=>"/tmp/mysincedbfile
3

Getting this errors, without sincedb_path

4

If you removed the sincedb_path option then the file input will log an INFO message saying what sincedb_path it is using. Verify that the user running logstash has write access to it.

5

How can change the logstash running user?

After removing sincedb_path the below log is printed

[INFO ][logstash.inputs.file     ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/aarch64/logstash/data/plugins/inputs/file/.sincedb_22914414f9f7324c1744d3b8907f8420",

Navigated to the above sincedb_path and ran ls -ltrh command

cd /aarch64/logstash/data/plugins/inputs/
ls -ltrh

drwxr-xr-x 2 user1 user1 4.0K Sep 24 16:36 file

where as logstash plain

-rw-rw-r-- 1 user1   qlrm   236K Sep 24 16:36 logstash-plain.log
6

One more thing. 7.9.1 is working perfectly in _x86 machines only in arm it's throwing exceptions

7

This feels like a bug to me. Only privileged processes can change the ownership of a file. Calling chown makes no sense to me. I could understand trying to chgrp (and ignoring a failure) but not chown.

If you have some systems where it works and some where it does not then I would guess the ones where it works are choosing to use the non-atomic write function.

The non-atomic write is used on Windows, or if the sincedb_path points to something that is neither a character nor a block device. Frankly I cannot think of a circumstance where that would occur, but you seem to have hit it.

8

Is it possible to disable the sincedb functionality ?

9

No. The in-memory sincedb is always maintained. You can set

sincedb_path => "/dev/null"

but it will still write the in-memory db to /dev/null, and it may still choose to use the atomic write for that. In that case it would chown /dev/null

If I were you I would try pointing sincedb_path at every file system you have available, and see if one of them chooses the non-atomic write. Local disk, RAM disk, NFS mount, /dev/null...

I see you are on aarch64. A user had the same problem a couple of years ago. No solution was found. I am wondering if .blockdev? is broken in the ruby implementation.

10

Not seeing exceptions after setting sincedb_path => "/dev/null"
Want to check, will it create any other issues if we set the sincedb_path => "/dev/null"

11

The in-memory sincedb will not be persisted across restarts, so when logstash is restarted it may re-read the files. That may or may not be a problem, depending on your use case.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.