Cannot change log format with pipeline config file, pipeline config file is not getting read

Elastic Stack Logstash
1

Greetings,
First time posting here,

elasticsearch 8.9
rhel 7
I am getting logs from jenkins jobs using logstash plugin with no issues
the only mime type that seems to work is "application/json"
If i try any other type, they will not send

  1. I cannot change the format of the logs I am receiving in any way
  2. The config file for the pipelines is not getting read no matter how I change them
  3. The logs I capture always seem to be in an arraylist format like this:
 "message": [
    "[...truncated 937 lines...]",
    "[2023-07-17T19:08:19.681Z] [INFO] Downloaded from repo.......",
    "[2023-07-17T19:08:19.681Z] [INFO] Downloading from repo......",
    "[2023-07-17T19:08:21.599Z] [Pipeline] logstashSend"
  ],
  1. Since my conf files arent being read I have attempted to edit my pipeline in the "Ingest pipelines" section but I get an error saying that arraylist cannot be converted to string. I have used json, plain, line plugins but i get the same error no matter what.

I use "systemctl start logstash" for startup so I cant use bin/logstash -f jenkins-pipeline.conf to point to my config file

I have edited the logstash.service contents to remove ""--path.settings" "/etc/logstash"from the execstart line:
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash

pipelines.yml contents:
- pipeline.id: jenkins-log
  path.config: "/etc/logstash/conf.d/jenkins-pipeline.conf"

jenkins-pipeline.conf placed in /etc/logstash/conf.d     
contents:

input {
tcp {
port => 10514
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "jenkins-log"
}
stdout { codec => plain }
}

for codec I have tried json,line,plain and others but the format never changes
I have placed my pipelines.yml in my etc/logstash folder as shown below

sudo ls -ltr /etc/logstash
total 52
-rw-------. 1 logstash logstash 1696 Jul 18 22:13 startup.options
-rw-r--r--. 1 logstash logstash 342 Jul 18 22:13 logstash-sample.conf
-rw-r--r--. 1 logstash logstash 7437 Jul 18 22:13 log4j2.properties
-rw-r--r--. 1 logstash logstash 1833 Jul 18 22:13 jvm.options.25834.2023-08-01@20:48:18~
-rwxrwxrwx. 1 logstash logstash 15153 Jul 31 21:19 logstash.yml
-rw-r--r--. 1 logstash logstash 159 Aug 1 20:48 ls.java.security
-rw-r--r--. 1 logstash logstash 58 Aug 1 20:48 java.security.options
-rw-r--r--. 1 logstash logstash 1926 Aug 1 20:48 jvm.options
drwxr-xr-x. 2 logstash logstash 35 Aug 2 00:12 conf.d
-rwxrwxrwx. 1 logstash logstash 86 Aug 2 16:16 pipelines.yml
drwxrwxrwx. 2 logstash logstash 6 Aug 2 20:12 tmp

Thanks
2

Do you have anything in Logstash logs?

Any reason for this? I think you need to have the path.settings to tell the logstash process where the configurations file are, not sure how it will behave without this, what do you have in the logs when you removed this and started logstash?

Since this is an array, a simple split filter in your logstash pipeline would split every one of these messages into a different event.

Just adding this to your pipeline would split the array into multiple events.

filter {
 split {
   field => "message"
 }
}
3

I read this in a post for grabbing the configs as a parameter when giving the startup command
I think this was part of the method for starting with the /bin/logstash command that I am not using.

4

I have not checked the logs I only followed the guide and see no results. I will check.

5

Following up with logs.
My message error went away when I added [message].
new pipeline config:

input {
tcp {
  id => "jenkins-log-pipeline"
  port => 10514
  }
}

filter {
 split {
   field =>  "[message]"
       }
}

output {
  elasticsearch {
    hosts => "127.0.0.1:9200"
    index => "jenkins-logging"
  }
  stdout { }
}

My pipeline created in my config file is named jenkins-log so this is the result of the searches of jenkins-logging, jenkins-log-pipeline and jenkins-log:

sudo cat /var/log/logstash/logstash-plain.log | grep jenkins-logging
[2023-08-03T23:15:49,689][INFO ][logstash.outputs.elasticsearch][jenkins-log] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"jenkins-logging"}

 sudo cat /var/log/logstash/logstash-plain.log | grep jenkins-log-pipeline
[2023-08-03T22:55:01,964][INFO ][logstash.inputs.tcp      ][jenkins-log][jenkins-log-pipeline] Starting tcp input listener {:address=>"0.0.0.0:10514", :ssl_enable=>false}

sudo cat /var/log/logstash/logstash-plain.log | grep jenkins-log
2023-08-03T23:15:49,691][INFO ][logstash.outputs.elasticsearch][jenkins-log] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-08-03T23:15:49,706][INFO ][logstash.outputs.elasticsearch][jenkins-log] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2023-08-03T23:15:49,723][INFO ][logstash.javapipeline    ][jenkins-log] Starting pipeline {:pipeline_id=>"jenkins-log", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/jenkins-pipeline.conf"], :thread=>"#<Thread:0x4037e1dc@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-08-03T23:15:51,169][INFO ][logstash.javapipeline    ][jenkins-log] Pipeline Java execution initialization time {"seconds"=>1.43}
[2023-08-03T23:15:51,363][INFO ][logstash.javapipeline    ][jenkins-log] Pipeline started {"pipeline.id"=>"jenkins-log"}
[2023-08-03T23:15:51,364][INFO ][logstash.inputs.tcp      ][jenkins-log][jenkins-log-pipeline] Starting tcp input listener {:address=>"0.0.0.0:10514", :ssl_enable=>false}
[2023-08-03T23:15:51,384][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:"jenkins-lo "], :non_running_pipelines=>[]}

The logs in "Discovery" are exactly the same as before:

    "[2023-08-03T23:19:51.616Z] Started by user ....",
    "[2023-08-03T23:19:51.735Z] [Pipeline] Start of Pipeline",
    "[2023-08-03T23:19:52.040Z] [Pipeline] node",
6

Please share a sample message from Discovery, just expand one of the documents, go to the json tab and copy and share the entire json document.

7

The target field [message] is exactly the same as the other fields, an arraylist.

{
  "@buildTimestamp": [
    "2023-08-04T14:01:27.418Z"
  ],
  "@timestamp": [
    "2023-08-04T14:04:17.450Z"
  ],
  "@version": [
    1
  ],
  "data.buildDuration": [
    170017
  ],
  "data.buildHost": [
    "Jenkins"
  ],
  "data.buildHost.keyword": [
    "Jenkins"
  ],
  "data.buildLabel": [
    "master"
  ],
  "data.buildLabel.keyword": [
    "master"
  ],
  "data.buildNum": [
    99
  ],
  "data.buildVariables.ANSIBLE_VERSION": [
    ""
  ],
  "data.buildVariables.ANSIBLE_VERSION.keyword": [
    ""
  ],
  "data.buildVariables.BUILD_DISPLAY_NAME": [
    "#99"
  ],
  "data.buildVariables.BUILD_DISPLAY_NAME.keyword": [
    "#99"
  ],
  "data.buildVariables.BUILD_ID": [
    "99"
  ],
  "data.buildVariables.BUILD_ID.keyword": [
    "99"
  ],
  "data.buildVariables.BUILD_NUMBER": [
    "99"
  ],
  "data.buildVariables.BUILD_NUMBER.keyword": [
    "99"
  ],
  "data.buildVariables.BUILD_TAG": [
    "jenkinsjenkins-win-configure-99"
  ],
  "data.buildVariables.BUILD_TAG.keyword": [
    "jenkins-jenkins-win-configure-99"
  ],
  "data.buildVariables.BUILD_URL": [
    "https://.com/jobjenkins/job/win-configure/99/"
  ],
  "data.buildVariables.BUILD_URL.keyword": [
    "https://com/jobnkins/job/win-configure/99/"
  ],
  "data.buildVariables.CI": [
    "false"
  ],
  "data.buildVariables.CI.keyword": [
    "false"
  ],
  "data.buildVariables.CLASSPATH": [
    ""
  ],
  "data.buildVariables.CLASSPATH.keyword": [
    ""
  ],
  "data.buildVariables.GCP_PROMOTE": [
    "true"
  ],
  "data.buildVariables.GCP_PROMOTE.keyword": [
    "true"
  ],
  "data.buildVariables.HUDSON_HOME": [
    "/mnt/jenkins"
  ],
  "data.buildVariables.HUDSON_HOME.keyword": [
    "/mnt/jenkins"
  ],
  "data.buildVariables.HUDSON_SERVER_COOKIE": [
    "f78b065630717635"
  ],
  "data.buildVariables.HUDSON_SERVER_COOKIE.keyword": [
    "f78b065630717635"
  ],
  "data.buildVariables.HUDSON_URL": [
    "https://.com/"
  ],
  "data.buildVariables.HUDSON_URL.keyword": [
    "https://com/"
  ],
  "data.buildVariables.JENKINS_HOME": [
    "/mnt/jenkins"
  ],
  "data.buildVariables.JENKINS_HOME.keyword": [
    "/mnt/jenkins"
  ],
  "data.buildVariables.JENKINS_SERVER_COOKIE": [
    "f78b065630717635"
  ],
  "data.buildVariables.JENKINS_SERVER_COOKIE.keyword": [
    "f78b065630717635"
  ],
  "data.buildVariables.JENKINS_URL": [
    "https://com/"
  ],
  "data.buildVariables.JENKINS_URL.keyword": [
    "https://.com/"
  ],
  "data.buildVariables.JOB_BASE_NAME": [
    "win-configure"
  ],
  "data.buildVariables.JOB_BASE_NAME.keyword": [
    "win-configure"
  ],
  "data.buildVariables.JOB_DISPLAY_URL": [
    "https://job/win-configure/display/redirect"
  ],
  "data.buildVariables.JOB_DISPLAY_URL.keyword": [
    "https:///job/win-configure/display/redirect"
  ],
  "data.buildVariables.JOB_NAME": [
    "enkins/win-configure"
  ],
  "data.buildVariables.JOB_NAME.keyword": [
    "enkins/win-configure"
  ],
  "data.buildVariables.JOB_URL": [
    "https://c/job/win-configure/"
  ],
  "data.buildVariables.JOB_URL.keyword": [
    "https:///job/win-configure/"
  ],
  "data.buildVariables.PROMOTION_URL": [
    "https://gcp.promote.repository.com"
  ],
  "data.buildVariables.PROMOTION_URL.keyword": [
    "https://gcp.promote.repository.com"
  ],
  "data.buildVariables.REPOSITORY": [
    "repository..com"
  ],
  "data.buildVariables.REPOSITORY.keyword": [
    "repositorycom"
  ],
  "data.buildVariables.RUN_ARTIFACTS_DISPLAY_URL": [
    "https:///job/win-configure/99/display/redirect?page=artifacts"
  ],
  "data.buildVariables.RUN_ARTIFACTS_DISPLAY_URL.keyword": [
    "https://.com/job/-jenkins/job/win-configure/99/display/redirect?page=artifacts"
  ],
  "data.buildVariables.RUN_CHANGES_DISPLAY_URL": [
    "https://com/jobenkins/job/win-configure/99/display/redirect?page=changes"
  ],
  "data.buildVariables.RUN_CHANGES_DISPLAY_URL.keyword": [
    "https:///job/win-configure/99/display/redirect?page=changes"
  ],
  "data.buildVariables.RUN_DISPLAY_URL": [
    "https://com/jobjenkins/job/win-configure/99/display/redirect"
  ],
  "data.buildVariables.RUN_DISPLAY_URL.keyword": [
    "https://.com/jobenkins/job/win-configure/99/display/redirect"
  ],
  "data.buildVariables.RUN_TESTS_DISPLAY_URL": [
    "https://com/jobjenkins/job/win-configure/99/display/redirect?page=tests"
  ],
  "data.buildVariables.RUN_TESTS_DISPLAY_URL.keyword": [
    "https://com/jobnkins/job/win-configure/99/display/redirect?page=tests"
  ],
  "data.buildVariables.RUN_VERACODE": [
    "false"
  ],
  "data.buildVariables.RUN_VERACODE.keyword": [
    "false"
  ],
  "data.buildVariables.SKIP_SONAR": [
    "true"
  ],
  "data.buildVariables.SKIP_SONAR.keyword": [
    "true"
  ],
  "data.displayName": [
    "#99"
  ],
  "data.displayName.keyword": [
    "#99"
  ],
  "data.fullDisplayName": [
    "jenkins » win-configure #99"
  ],
  "data.fullDisplayName.keyword": [
    "jenkins » win-configure #99"
  ],
  "data.fullProjectName": [
    "jenkins/win-configure"
  ],
  "data.fullProjectName.keyword": [
    "nkins/win-configure"
  ],
  "data.id": [
    "99"
  ],
  "data.id.keyword": [
    "99"
  ],
  "data.projectName": [
    "win-configure"
  ],
  "data.projectName.keyword": [
    "win-configure"
  ],
  "data.rootBuildNum": [
    99
  ],
  "data.rootFullProjectName": [
    "/win-configure"
  ],
  "data.rootFullProjectName.keyword": [
    "/win-configure"
  ],
  "data.rootProjectDisplayName": [
    "#99"
  ],
  "data.rootProjectDisplayName.keyword": [
    "#99"
  ],
  "data.rootProjectName": [
    "win-configure"
  ],
  "data.rootProjectName.keyword": [
    "win-configure"
  ],
  "data.url": [
    "job/job/win-configure/99/"
  ],
  "data.url.keyword": [
    "job//job/win-configure/99/"
  ],
  "message": [
    "[2023-08-04T14:01:27.432Z] Started by user",
    "[2023-08-04T14:01:27.601Z] [Pipeline] Start of Pipeline",
    "[2023-08-04T14:01:27.739Z] [Pipeline] node",
    "[2023-08-04T14:01:42.746Z] Still waiting to schedule task",
    "[2023-08-04T14:01:42.746Z] ‘agent-s6hd8c’ is offline",
    "[2023-08-04T14:04:10.866Z] Running ons-agent-s6hd8c in /tmp/workspace/jenkins/win-configure",
    "[2023-08-04T14:04:10.873Z] [Pipeline] {",
    "[2023-08-04T14:04:10.920Z] [Pipeline] stage",
    "[2023-08-04T14:04:10.926Z] [Pipeline] { (Set Ansible Version)",
    "[2023-08-04T14:04:10.971Z] [Pipeline] script",
    "[2023-08-04T14:04:10.977Z] [Pipeline] {",
    "[2023-08-04T14:04:11.021Z] [Pipeline] httpRequest",
    "[2023-08-04T14:04:11.253Z] HttpMethod: GET",
    "[2023-08-04T14:04:11.253Z] URL: https://gcp.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=com.&name=-ansible&version=",
    "[2023-08-04T14:04:11.341Z] Using authentication: -repository-token",
    "[2023-08-04T14:04:11.841Z] Sending request to url: https://.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=com&name=nkins-win-ansible&version=",
    "[2023-08-04T14:04:12.299Z] Response Code: HTTP/1.1 200 OK",
    "[2023-08-04T14:04:12.345Z] Success: Status code 200 is in the accepted range: 100:399",
    "[2023-08-04T14:04:12.364Z] [Pipeline] readJSON",
    "[2023-08-04T14:04:12.407Z] [Pipeline] echo",
    "[2023-08-04T14:04:12.413Z] all matching versions=[1.0.0-2-g26dc582]",
    "[2023-08-04T14:04:12.424Z] [Pipeline] echo",
    "[2023-08-04T14:04:12.430Z] selected version=1.0.0-2-g26dc582",
    "[2023-08-04T14:04:12.441Z] [Pipeline] sh",
    "[2023-08-04T14:04:12.747Z] + gcloud compute project-info add-metadata --metadata -win-ansible-version=1.0.0-2-g26dc582",
    "[2023-08-04T14:04:14.129Z] No change requested; skipping update for [sab-dev5697].",
    "[2023-08-04T14:04:14.417Z] [Pipeline] sleep",
    "[2023-08-04T14:04:14.423Z] Sleeping for 3 sec",
    "[2023-08-04T14:04:17.442Z] [Pipeline] logstashSend"
  ],
  "message.keyword": [
    "[2023-08-04T14:01:27.432Z] Started by user ",
    "[2023-08-04T14:01:27.601Z] [Pipeline] Start of Pipeline",
    "[2023-08-04T14:01:27.739Z] [Pipeline] node",
    "[2023-08-04T14:01:42.746Z] Still waiting to schedule task",
    "[2023-08-04T14:01:42.746Z] ‘-agent-s6hd8c’ is offline",
    "[2023-08-04T14:04:10.866Z] Running on -jenkins-agent-s6hd8c in /tmp/workspace//win-configure",
    "[2023-08-04T14:04:10.873Z] [Pipeline] {",
    "[2023-08-04T14:04:10.920Z] [Pipeline] stage",
    "[2023-08-04T14:04:10.926Z] [Pipeline] { (Set Ansible Version)",
    "[2023-08-04T14:04:10.971Z] [Pipeline] script",
    "[2023-08-04T14:04:10.977Z] [Pipeline] {",
    "[2023-08-04T14:04:11.021Z] [Pipeline] httpRequest",
    "[2023-08-04T14:04:11.253Z] HttpMethod: GET",
    "[2023-08-04T14:04:11.253Z] URL: https://repository.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=comhen&name=win-ansible&version=",
    "[2023-08-04T14:04:11.341Z] Using authentication: repository-token",
    "[2023-08-04T14:04:11.841Z] Sending request to url: https://gcp.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=com.&name=-win-ansible&version=",
    "[2023-08-04T14:04:12.299Z] Response Code: HTTP/1.1 200 OK",
    "[2023-08-04T14:04:12.345Z] Success: Status code 200 is in the accepted range: 100:399",
    "[2023-08-04T14:04:12.364Z] [Pipeline] readJSON",
    "[2023-08-04T14:04:12.407Z] [Pipeline] echo",
    "[2023-08-04T14:04:12.413Z] all matching versions=[1.0.0-2-g26dc582]",
    "[2023-08-04T14:04:12.424Z] [Pipeline] echo",
    "[2023-08-04T14:04:12.430Z] selected version=1.0.0-2-g26dc582",
    "[2023-08-04T14:04:12.441Z] [Pipeline] sh",
    "[2023-08-04T14:04:12.747Z] + gcloud compute project-info add-metadata --metadata -win-ansible-version=1.0.0-2-g26dc582",
    "[2023-08-04T14:04:14.129Z] No change requested; skipping update for [ore-5697].",
    "[2023-08-04T14:04:14.417Z] [Pipeline] sleep",
    "[2023-08-04T14:04:14.423Z] Sleeping for 3 sec",
    "[2023-08-04T14:04:17.442Z] [Pipeline] logstashSend"
  ],
  "source": [
    "jenkins"
  ],
  "source_host": [
    "https:/.com/"
  ],
  "source_host.keyword": [
    "https://.com/"
  ],
  "source.keyword": [
    "jenkins"
  ],
  "_id": "CirdwIkBDKSsBQJEBEiy",
  "_index": "jenkins-log",
  "_score": null
}
8

my response was hidden for a little while, but this is the contect with references to infrastructure removed

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.