Unable to read logs files

Elastic Stack Logstash
1

Good morning,

I'm trying to read log files from a concrete path. My configuration is the following

input {
        file {
              path => "/path/proxy/access2.log"
              start_position => "beginning"
        }
}
filter {
        grok {

match => [ "message","%{NUMBER:timestamp}%{SPACE}%{NUMBER:duration}\s%{IP:client_address}\s%{WORD:cache_result}/%{POSINT:status_code}\s%{NUMBER:bytes}\s%{WORD:request_method}\s%{NOTSPACE:url}\s%{NOTSPACE:user}\s%{WORD:hierarchy_code}/(%{IP:dst_addrss})?%{NOTSPACE:server}\s%{NOTSPACE:content_type}" ]
                }
        date {
                match => [ "timestamp", "UNIX" ]
                remove_field => [ "timestamp" ]
             }

        }

output {
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "squid"
  }
stdout {  }
}

I've set the debug log in logstash and appears the following log

[2021-05-25T08:07:02,420][INFO ][logstash.javapipeline    ][squid] Pipeline Java execution initialization time {"seconds"=>0.74}
[2021-05-25T08:07:02,633][TRACE][logstash.inputs.file     ][squid] Registering file input {:path=>["/home/jallende/proxy/access2.log"]}
[2021-05-25T08:07:05,928][INFO ][logstash.inputs.file     ][squid] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67", :path=>["/home/jallende/proxy/access2.log"]}
[2021-05-25T08:07:05,938][INFO ][logstash.javapipeline    ][squid] Pipeline started {"pipeline.id"=>"squid"}
[2021-05-25T08:07:05,940][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:05,947][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>"squid", :thread=>"#<Thread:0x6cf5c4d9 sleep>"}
[2021-05-25T08:07:05,950][TRACE][logstash.agent           ] Converge results {:success=>true, :failed_actions=>[], :successful_actions=>["id: reglas, action_type: LogStash::PipelineAction::Create", "id: vpn, action_type: LogStash::PipelineAction::Create", "id: arcsight, action_type: LogStash::PipelineAction::Create", "id: eps, action_type: LogStash::PipelineAction::Create", "id: squid, action_type: LogStash::PipelineAction::Create"]}
[2021-05-25T08:07:05,969][INFO ][logstash.agent           ] Pipelines running {:count=>5, :running_pipelines=>[:reglas, :vpn, :arcsight, :eps, :squid], :non_running_pipelines=>[]}
[2021-05-25T08:07:05,973][INFO ][filewatch.observingtail  ][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] START, creating Discoverer, Watch with file and sincedb collections
[2021-05-25T08:07:05,988][DEBUG][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] open: reading from /var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67
[2021-05-25T08:07:05,990][TRACE][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] open: count of keys read: 0
[2021-05-25T08:07:06,006][TRACE][filewatch.discoverer     ][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] discover_files {:count=>0}
[2021-05-25T08:07:07,023][DEBUG][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] writing sincedb (delta since last write = 1621922827)
[2021-05-25T08:07:07,027][TRACE][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] sincedb_write: /var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67 (time = 2021-05-25 08:07:07 +0200)
[2021-05-25T08:07:10,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:15,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:20,044][TRACE][filewatch.discoverer     ][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] discover_files {:count=>0}
[2021-05-25T08:07:20,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:22,045][DEBUG][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] writing sincedb (delta since last write = 15)
[2021-05-25T08:07:22,046][TRACE][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] sincedb_write: /var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67 (time = 2021-05-25 08:07:22 +0200)
[2021-05-25T08:07:25,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:30,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:35,049][TRACE][filewatch.discoverer     ][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] discover_files {:count=>0}
[2021-05-25T08:07:35,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:37,050][DEBUG][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] writing sincedb (delta since last write = 15)
[2021-05-25T08:07:37,050][TRACE][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] sincedb_write: /var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67 (time = 2021-05-25 08:07:37 +0200)
[2021-05-25T08:07:40,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:45,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:50,053][TRACE][filewatch.discoverer     ][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] discover_files {:count=>0}
[2021-05-25T08:07:50,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:07:52,054][DEBUG][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] writing sincedb (delta since last write = 15)
[2021-05-25T08:07:52,054][TRACE][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] sincedb_write: /var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67 (time = 2021-05-25 08:07:52 +0200)
[2021-05-25T08:07:55,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:08:00,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:08:05,057][TRACE][filewatch.discoverer     ][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] discover_files {:count=>0}
[2021-05-25T08:08:05,939][DEBUG][org.logstash.execution.PeriodicFlush][squid] Pushing flush onto pipeline.
[2021-05-25T08:08:07,058][DEBUG][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] writing sincedb (delta since last write = 15)
[2021-05-25T08:08:07,058][TRACE][filewatch.sincedbcollection][squid][ce608fea0833a3776a6f3a4c4cd5d54e390a7df1f53b075797fe3227dd8c7923] sincedb_write: /var/lib/logstash/plugins/inputs/file/.sincedb_5154ac82e8774c7923b7ced1fd141b67 (time = 2021-05-25 08:08:07 +0200)

The sincedb file is empty and the index information in kibana is emtpy too , any idea?

2

The file input is not finding a file that matches "/path/proxy/access2.log"

3

sorry, I've forget to answer, the problem resides in the permission

4

I think the problem is in the permission but I've other problem cos if i run in this way /usr/share/logstash/bin/logstash --debug -f /etc/logstash/squid.conf its works but if i run in service logstash start the log file isnt read. Any idea?

regards

5

Use ps to see which user the service is running as. Check each element of the path to make sure that the user has x access to the directories and r access to the log.

6

Process as logstash
folders and files has 777 permission
drwxrwxrwx. 2 user user 24 may 26 07:04 proxy
-rwxrwxrwx. 1 user user 43974656 may 26 07:03 access.log

any idea?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.