Logstash beginner configuration issues

Marie-Laure_Aix · January 27, 2020, 3:55pm

Hi there,

I am a complete beginner with Logstash and I have issues configurating it. I am getting this error message :

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-01-27T16:51:43,379][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:in validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in block in validate_all'", "org/jruby/RubyHash.java:1417:in each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:284:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in '"]}
[2020-01-27T16:51:43,483][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Is anyone familiar with this kind of error message ?

Many thanks

Marie-Laure

Badger · January 27, 2020, 4:21pm

What is unclear about that error message?

Marie-Laure_Aix · January 27, 2020, 9:00pm

Basically everything I don't understand how to solve the whole bunch of issues in this error message !! I am a disaster

Badger · January 27, 2020, 10:08pm

You need to chmod the /var/lib/logstash/queue directory so that it is writeable by the user who is running logstash.

Marie-Laure_Aix · January 28, 2020, 8:21am

Hi Badger,

I tried chmod doing this :

$sudo chmod -R 755 /var/lib/logstash/queue

But it did not work. Do you think this is the correct way ?

Many thanks

Badger · January 28, 2020, 3:30pm

That would work if the user who is running logstash owns the /var/lib/logstash/queue directory. Personally I have the directory group writeable and add myself to the logstash group, then make the directory 775.

benoitdupont · January 28, 2020, 6:57pm

Bonjour Marie-Laure,

I installed logstash on Ubuntu via the software installer, and it created a logstash user, and set the permissions accordingly here:

ben@Ben:/usr/share/logstash$ ls -l /var/lib/logstash
drwxr-xr-x 2 logstash logstash 4096 Jan 23 16:32 queue

When I launch logstash, I use sudo -u logstash bin/logstash ... and it is allowed then to write on /var/lib/logstash

If this doesn't work, can you can give more information on the user used to launch logstash, and display the permissions of the /var/lib/logstash folder ?

Cheers,

Ben

Marie-Laure_Aix · January 29, 2020, 8:03am

Salut Benoît,

Thanks for your message. I tried to reinstall the software with Synaptic but I am still getting the same error message ...

Do you know the command line used to know which user is used to launch logstash ?

Regarding the permissions I am getting this message :

drwxr-xr-x 2 logstash logstash 4096 janv. 27 15:07 dead_letter_queue
drwxr-xr-x 2 aixm logstash 4096 janv. 27 15:07 queue
-rw-r--r-- 1 logstash logstash 36 janv. 27 15:07 uuid

Cheers,

ML

Bandana_Mahal · January 29, 2020, 8:58am

Hello,

I am a beginner in logstash and want to parse xml to json using logstash filter plugin.
I need my json output removing few tags and few attributes from some tags.

Below is my xml

<?xml version="1.0" encoding="UTF-8" ?>
<Model_3dxml xmlns="http://www.3ds.com/xsd/3DXML" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Header>
    <SchemaVersion>4.3</SchemaVersion>
    <Title>PRODUCT</Title>
    <Author>mchi</Author>
    <Generator>Dassault Systemes V6</Generator>
    <Created>2020-01-24</Created>
  </Header>
  <ProductStructure root="1" xmlns:_3DShape="http://www.3ds.com/xsd/3DXML/PRODUCT/3DShape" xsi:schemaLocation="http://www.3ds.com/xsd/3DXML/PRODUCT/3DShape 3DShape.xsd ">
    <Reference3D xsi:type="Reference3DType" id="1" name="prd-43415814-00025623">
      <C_created>1579877063</C_created>
      <C_modified>1579877063</C_modified>
      <PLM_ExternalID>prd-43415814-00025623</PLM_ExternalID>
      <V_Name>Produit physique00025623</V_Name>
      <V_nature>2</V_nature>
      <V_version>A</V_version>
      <V_maturity>IN_WORK</V_maturity>
    </Reference3D>
    <ReferenceRep xsi:type="_3DShape:_AddedPrefix_3DShape" id="3" name="3sh-43415814-00022160" format="UVR" associatedFile="urn:3DXML:3sh-43415814-00022160_2_46ea3bc6_4004_5e2b0dbf_a7e7.3DRep" version="1.0">
      <C_created>1579877062</C_created>
      <C_modified>1579877063</C_modified>
      <PLM_ExternalID>3sh-43415814-00022160</PLM_ExternalID>
      <V_discipline>Design</V_discipline>
      <V_Name>Forme 3D00022160</V_Name>
      <V_nature>2</V_nature>
      <V_version>A</V_version>
      <V_maturity>IN_WORK</V_maturity>
    </ReferenceRep>
    <InstanceRep xsi:type="InstanceRepType" id="2" name="3sh-43415814-00022160.1">
      <C_created>1579877063</C_created>
      <C_modified>1579877063</C_modified>
      <PLM_ExternalID>3sh-43415814-00022160.1</PLM_ExternalID>
      <V_nature>2</V_nature>
      <IsAggregatedBy>1</IsAggregatedBy>
      <IsInstanceOf>3</IsInstanceOf>
    </InstanceRep>
  </ProductStructure>
</Model_3dxml>

And here is my filter

input {
file {
path => "C:/Users/bmahal/Desktop/MyDocuments/IDEE Project/Dataset/Produit physique00021849 A.1/PRODUCT.3dxml"
start_position => "beginning"
sincedb_path => "NUL"
mode => read
codec => multiline {
#pattern => "^<?Model_3dxml.*>"
pattern => "^\s<Model_3dxml"
negate => "true"
what => "previous"
auto_flush_interval => 3
max_lines => 3000
}
}
}

filter {
xml {
remove_namespaces => "true"
source => "message"
store_xml => "true"
target => "jsondata"
force_array => "false"

}	
mutate {
	remove_field => [ "message" ]
}

}

output {
file {
path => "C:/Users/bmahal/Desktop/MyDocuments/Softwares/logstash-7.5.2/output.json"
}
stdout {
codec => rubydebug
}
}

I want to remove "<PLM_ExternalID>prd-43415814-00025623</PLM_ExternalID>" and attribute xmlns:_3DShape. Your help will be appreciated.

benoitdupont · January 29, 2020, 11:52am

Hello, Synaptic creates a 'logstash' user, then set all logstash installation directory permissions with group 'logstash' and user 'logstash'.

The first possibility, as Badger says, is to add yourself to the group 'logstash':
sudo usermod -a -G logstash aixm, then check if it's ok by running groups, 'logstash' should be listed (note that you should restart your session first).

The second possibility is to specify the user that run logstash, which replies to your question:

Blockquote
Do you know the command line used to know which user is used to launch logstash ?

That's what you input: sudo -u logstash bin/logstash, the 'sudo -u logstash' will then use the user 'logstash' to run logstash.

Beware that your queue folder is on user permission to aix, it would be better to chown logstash queue it to logstash again with this solution to be consistent.

If you go for the first solution, at least try to be consistent and adapt all folders and files to be owned by 'aix'.

If someone can validate or argument why one approach is better than the otter, it would be interesting to know . The second approach seems more secure because you only have the logstash rights for the duration of the execution, but is more verbose to type-in (maybe we drift off-topic here and it should be discussed on a separate post).

Marie-Laure_Aix · January 29, 2020, 2:00pm

Thanks Benoit ! I tried the first option and it worked !

system · February 26, 2020, 2:00pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.