FSCrawler 7.2.7, Windows service, debug and trace logs

When running fscrawler 7.2.7 as a service (Windows server 2016) should it be possible to see debug or trace logs by adding the --debug or --trace flag?

We are using nssm v2.24 to generate the service.

I successfully added the --debug or --trace flag to a batch file that was called by a Scheduled task (configured to run on system startup after x minutes) but we would like to use a Windows Service.

(batch file details to follow.)

I'm wondering if you can edit the config/log4j2.xml file and activate a trace level in it instead of using the CLI options... Have a look at:

   <Properties>
      <!-- If you want to change the log level for fscrawler.log file -->
      <Property name="LOG_LEVEL">info</Property>
      <!-- If you want to change the log level for documents.log file -->
      <Property name="DOC_LEVEL">info</Property>
      <!-- If you want to change the output dir for logs -->
      <Property name="LOG_DIR">logs</Property>
   </Properties>

This is my fscrawlerRunner_V032.bat

I think it is picking up the log4j_v032.xml file (line 3) as I no longer get:

ERROR StatusLogger Reconfiguration failed: No configuration found for '73d16e93' at 'null' in 'null'

<--------start fscrawlerRunner_V032.bat ------------->

> set JAVA_HOME=c:\Program Files\elasticsearch-7.10.2\jdk
> set FS_JAVA_OPTS=-Xmx2g -Xms2g
> set FS_JAVA_OPTS="-Dlog4j.configurationFile=file:C:\Progra~1\fscrawler-es7-2.7\data2\testv003\log4j2_v032.xml" 
> c:\Progra~1\fscrawler-es7-2.7\bin\fscrawler.bat --config_dir c:\Progra~1\fscrawler-es7-2.7\data2 testv003 >> c:\Progra~1\fscrawler-es7-2.7\logs\fscrawler.log 2>&1

<----------end --------->

When I run this as a service (using nssm.exe)
C:\Program Files\fscrawler-es7-2.7\logs\fscrawler.log contains the fscrawler splash screen

..\logs\documents.log
is empty ie not a line per document indexed
_status.json is updated with the number of indexed files.

Please could give some pointers on how to configure the default log4j2.xml file

eg can you point to c:\tmp as the LOG_DIR?
What is sys:LOG_DIR on a Windows machine?
Is it C:\Progra~1\fscrawler-es7-2.7\logs by default ?

    <?xml version="1.0" encoding="UTF-8"?>
    <Configuration status="fatal" monitorInterval="30">
       <Properties>
          <!-- If you want to change the log level for fscrawler.log file -->
          <Property name="LOG_LEVEL">trace</Property>
          <!-- If you want to change the log level for documents.log file -->
          <Property name="DOC_LEVEL">trace</Property>
          <!-- If you want to change the output dir for logs -->
          <Property name="LOG_DIR">log</Property>
       </Properties>

       <Appenders>
          <Console name="Console" target="SYSTEM_OUT" follow="true">
             <PatternLayout pattern="%m%n"/>
          </Console>

          <RollingFile name="RollingFile" fileName="${sys:LOG_DIR}/fscrawler.log"
                       filePattern="${sys:LOG_DIR}/fscrawler-%d{yyyy-MM-dd}-%i.log.gz">
             <PatternLayout pattern="%d{ABSOLUTE} %highlight{%-5p} [%c{1.}] %m%n"/>
             <Policies>
                <OnStartupTriggeringPolicy />
                <SizeBasedTriggeringPolicy size="20 MB" />
                <TimeBasedTriggeringPolicy />
             </Policies>
             <DefaultRolloverStrategy max="7"/>
          </RollingFile>

          <RollingFile name="Documents" fileName="${sys:LOG_DIR}/documents.log"
                       filePattern="${sys:LOG_DIR}/documents-%d{yyyy-MM-dd}.log.gz">
             <PatternLayout pattern="%d [%highlight{%-5p}] %m%n"/>
             <Policies>
                <TimeBasedTriggeringPolicy />
             </Policies>
             <DefaultRolloverStrategy max="7"/>
          </RollingFile>
       </Appenders>
       <Loggers>
          <!-- This logger is used for the console -->
          <Logger name="fscrawler.console" level="info" additivity="false">
             <AppenderRef ref="Console" />
          </Logger>

          <!-- This logger is used to trace all information about documents -->
          <Logger name="fscrawler.document" level="${sys:DOC_LEVEL}" additivity="false">
             <AppenderRef ref="Documents" />
          </Logger>

          <!-- This logger is used to log FSCrawler code execution -->
          <Logger name="fr.pilato.elasticsearch.crawler.fs" level="${sys:LOG_LEVEL}" additivity="false">
             <AppenderRef ref="RollingFile" />
          </Logger>

          <!-- This logger is used to log 3rd party libs execution -->
          <Logger name="org.elasticsearch" level="warn" additivity="false">
             <AppenderRef ref="RollingFile" />
          </Logger>
          <Logger name="org.glassfish" level="warn" additivity="false">
             <AppenderRef ref="RollingFile" />
          </Logger>
          <Logger name="org.apache.tika.parser.ocr.TesseractOCRParser" level="error" additivity="false">
             <AppenderRef ref="RollingFile" />
          </Logger>
          <Logger name="com.gargoylesoftware" level="error" additivity="false">
             <AppenderRef ref="RollingFile"/>
          </Logger>

          <Root level="warn">
             <AppenderRef ref="RollingFile" />
          </Root>
       </Loggers>
    </Configuration>

Thanks to your advice I got the logging to work as I want.
1 line per document and low volumes for fscrawler.log

I specified the log path in LOG_DIR.

  <Properties>
      <!-- If you want to change the log level for fscrawler.log file -->
      <Property name="LOG_LEVEL">info</Property>
      <!-- If you want to change the log level for documents.log file -->
      <Property name="DOC_LEVEL">debug</Property>
      <!-- If you want to change the output dir for logs -->
      <Property name="LOG_DIR">c:\Progra~1\fscrawler-es7-2.7\data2\testv003\logs</Property>
   </Properties>

Looking back at my batch file fscrawlerRunner_V035.bat

I have 2 set FS_JAVA_OPTS. Does the 2nd overwrite the 1st?
How do I safely combine then or check their value?

set JAVA_HOME=c:\Program Files\elasticsearch-7.10.2\jdk
set FS_JAVA_OPTS=-Xmx2g -Xms2g
set FS_JAVA_OPTS="-Dlog4j.configurationFile=file:C:\Progra~1\fscrawler-es7-2.7\data2\testv003\log4j2_v035.xml" 
rem set FS_JAVA_OPTS="-DLOG_DIR=c:\Progra~1\fscrawler-es7-2.7\data2\testv003\logs -DLOG_LEVEL=trace -DDOC_LEVEL=debug"
rem "c:\Progra~1\fscrawler-es7-2.7\bin\fscrawler.bat" testv001 --config_dir C:\Progra~1\fscrawler-es7-2.7\data2\testv003
c:\Progra~1\fscrawler-es7-2.7\bin\fscrawler.bat --config_dir c:\Progra~1\fscrawler-es7-2.7\data2 testv003 >> c:\Progra~1\fscrawler-es7-2.7\logs\fscrawler.log 2>&1

Combining the two statements seems to work:

set FS_JAVA_OPTS=-Xmx2g -Xms2g "-Dlog4j.configurationFile=file:C:\Progra~1\fscrawler-es7-2.7\data2\testv003\log4j2_v035.xml" 
echo
 -Xmx2g -Xms2g "-Dlog4j.configurationFile=file:C:\Progra~1\fscrawler-es7-2.7\data2\testv003\log4j2_v035.xml"
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.