# Cannot change log format with pipeline config file, pipeline config file is not getting read
**URL:** https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081
**Category:** Logstash
**Created:** [August 3, 2023, 9:05pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081 "2023-08-03T21:05:05Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Jenkins-Jobs](https://discuss.elastic.co/u/Jenkins-Jobs)
#### Post date: [August 3, 2023, 9:05pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/1 "2023-08-03T21:05:05Z")
Greetings,
First time posting here,
elasticsearch 8.9
rhel 7
I am getting logs from jenkins jobs using logstash plugin with no issues
the only mime type that seems to work is "application/json"
If i try any other type, they will not send
1. I cannot change the format of the logs I am receiving in any way
2. The config file for the pipelines is not getting read no matter how I change them
3. The logs I capture always seem to be in an arraylist format like this:
```auto
"message": [
"[...truncated 937 lines...]",
"[2023-07-17T19:08:19.681Z] [INFO] Downloaded from repo.......",
"[2023-07-17T19:08:19.681Z] [INFO] Downloading from repo......",
"[2023-07-17T19:08:21.599Z] [Pipeline] logstashSend"
],
```
1. Since my conf files arent being read I have attempted to edit my pipeline in the "Ingest pipelines" section but I get an error saying that arraylist cannot be converted to string. I have used json, plain, line plugins but i get the same error no matter what.
I use "systemctl start logstash" for startup so I cant use bin/logstash -f jenkins-pipeline.conf to point to my config file
I have edited the logstash.service contents to remove ""--path.settings" "/etc/logstash"from the execstart line:
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash
```auto
pipelines.yml contents:
- pipeline.id: jenkins-log
path.config: "/etc/logstash/conf.d/jenkins-pipeline.conf"
```
```auto
jenkins-pipeline.conf placed in /etc/logstash/conf.d
contents:
```
input {
tcp {
port =\> 10514
}
}
output {
elasticsearch {
hosts =\> "127.0.0.1:9200"
index =\> "jenkins-log"
}
stdout { codec =\> plain }
}
```auto
for codec I have tried json,line,plain and others but the format never changes
I have placed my pipelines.yml in my etc/logstash folder as shown below
```
sudo ls -ltr /etc/logstash
total 52
-rw-------. 1 logstash logstash 1696 Jul 18 22:13 startup.options
-rw-r--r--. 1 logstash logstash 342 Jul 18 22:13 logstash-sample.conf
-rw-r--r--. 1 logstash logstash 7437 Jul 18 22:13 log4j2.properties
-rw-r--r--. 1 logstash logstash 1833 Jul 18 22:13 jvm.options.25834.2023-08-01@20:48:18~
-rwxrwxrwx. 1 logstash logstash 15153 Jul 31 21:19 logstash.yml
-rw-r--r--. 1 logstash logstash 159 Aug 1 20:48 ls.java.security
-rw-r--r--. 1 logstash logstash 58 Aug 1 20:48 java.security.options
-rw-r--r--. 1 logstash logstash 1926 Aug 1 20:48 jvm.options
drwxr-xr-x. 2 logstash logstash 35 Aug 2 00:12 conf.d
-rwxrwxrwx. 1 logstash logstash 86 Aug 2 16:16 pipelines.yml
drwxrwxrwx. 2 logstash logstash 6 Aug 2 20:12 tmp
```auto
Thanks
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 3, 2023, 9:41pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/2 "2023-08-03T21:41:47Z")
Do you have anything in Logstash logs?
> [@Jenkins-Jobs](#):
>
> I have edited the logstash.service contents to remove ""--path.settings" "/etc/logstash"from the execstart line:
> ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash
Any reason for this? I think you need to have the `path.settings ` to tell the logstash process where the configurations file are, not sure how it will behave without this, what do you have in the logs when you removed this and started logstash?
> [@Jenkins-Jobs](#):
>
> ```auto
> "message": [
> "[...truncated 937 lines...]",
> "[2023-07-17T19:08:19.681Z] [INFO] Downloaded from repo.......",
> "[2023-07-17T19:08:19.681Z] [INFO] Downloading from repo......",
> "[2023-07-17T19:08:21.599Z] [Pipeline] logstashSend"
> ]
>
> ```
Since this is an array, a simple [split filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html#_description_153) in your logstash pipeline would split every one of these messages into a different event.
Just adding this to your pipeline would split the array into multiple events.
```auto
filter {
split {
field => "message"
}
}
```
---
### Author:  [@Jenkins-Jobs](https://discuss.elastic.co/u/Jenkins-Jobs)
#### Post date: [August 3, 2023, 9:47pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/3 "2023-08-03T21:47:13Z")
> [@leandrojmp](#):
>
> `path.settings `
I read this in a post for grabbing the configs as a parameter when giving the startup command
I think this was part of the method for starting with the /bin/logstash command that I am not using.
---
### Author:  [@Jenkins-Jobs](https://discuss.elastic.co/u/Jenkins-Jobs)
#### Post date: [August 3, 2023, 9:50pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/4 "2023-08-03T21:50:02Z")
> [@leandrojmp](#):
>
> Do you have anything in Logstash logs
I have not checked the logs I only followed the guide and see no results. I will check.
---
### Author:  [@Jenkins-Jobs](https://discuss.elastic.co/u/Jenkins-Jobs)
#### Post date: [August 3, 2023, 11:33pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/5 "2023-08-03T23:33:48Z")
Following up with logs.
My message error went away when I added [message].
new pipeline config:
```auto
input {
tcp {
id => "jenkins-log-pipeline"
port => 10514
}
}
filter {
split {
field => "[message]"
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "jenkins-logging"
}
stdout { }
}
```
My pipeline created in my config file is named jenkins-log so this is the result of the searches of jenkins-logging, jenkins-log-pipeline and jenkins-log:
```auto
sudo cat /var/log/logstash/logstash-plain.log | grep jenkins-logging
[2023-08-03T23:15:49,689][INFO][logstash.outputs.elasticsearch][jenkins-log] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"jenkins-logging"}
sudo cat /var/log/logstash/logstash-plain.log | grep jenkins-log-pipeline
[2023-08-03T22:55:01,964][INFO][logstash.inputs.tcp][jenkins-log][jenkins-log-pipeline] Starting tcp input listener {:address=>"0.0.0.0:10514", :ssl_enable=>false}
```
```auto
sudo cat /var/log/logstash/logstash-plain.log | grep jenkins-log
2023-08-03T23:15:49,691][INFO][logstash.outputs.elasticsearch][jenkins-log] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-08-03T23:15:49,706][INFO][logstash.outputs.elasticsearch][jenkins-log] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2023-08-03T23:15:49,723][INFO][logstash.javapipeline][jenkins-log] Starting pipeline {:pipeline_id=>"jenkins-log", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/jenkins-pipeline.conf"], :thread=>"#"}
[2023-08-03T23:15:51,169][INFO][logstash.javapipeline][jenkins-log] Pipeline Java execution initialization time {"seconds"=>1.43}
[2023-08-03T23:15:51,363][INFO][logstash.javapipeline][jenkins-log] Pipeline started {"pipeline.id"=>"jenkins-log"}
[2023-08-03T23:15:51,364][INFO][logstash.inputs.tcp][jenkins-log][jenkins-log-pipeline] Starting tcp input listener {:address=>"0.0.0.0:10514", :ssl_enable=>false}
[2023-08-03T23:15:51,384][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:"jenkins-lo "], :non_running_pipelines=>[]}
```
The logs in "Discovery" are exactly the same as before:
```auto
"[2023-08-03T23:19:51.616Z] Started by user ....",
"[2023-08-03T23:19:51.735Z] [Pipeline] Start of Pipeline",
"[2023-08-03T23:19:52.040Z] [Pipeline] node",
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 4, 2023, 2:28am UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/6 "2023-08-04T02:28:57Z")
> [@Jenkins-Jobs](#):
>
> The logs in "Discovery" are exactly the same as before
Please share a sample message from Discovery, just expand one of the documents, go to the json tab and copy and share the entire json document.
---
### Author:  [@Jenkins-Jobs](https://discuss.elastic.co/u/Jenkins-Jobs)
#### Post date: [August 4, 2023, 2:37pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/7 "2023-08-04T14:37:45Z")
The target field [message] is exactly the same as the other fields, an arraylist.
```auto
{
"@buildTimestamp": [
"2023-08-04T14:01:27.418Z"
],
"@timestamp": [
"2023-08-04T14:04:17.450Z"
],
"@version": [
1
],
"data.buildDuration": [
170017
],
"data.buildHost": [
"Jenkins"
],
"data.buildHost.keyword": [
"Jenkins"
],
"data.buildLabel": [
"master"
],
"data.buildLabel.keyword": [
"master"
],
"data.buildNum": [
99
],
"data.buildVariables.ANSIBLE_VERSION": [
""
],
"data.buildVariables.ANSIBLE_VERSION.keyword": [
""
],
"data.buildVariables.BUILD_DISPLAY_NAME": [
"#99"
],
"data.buildVariables.BUILD_DISPLAY_NAME.keyword": [
"#99"
],
"data.buildVariables.BUILD_ID": [
"99"
],
"data.buildVariables.BUILD_ID.keyword": [
"99"
],
"data.buildVariables.BUILD_NUMBER": [
"99"
],
"data.buildVariables.BUILD_NUMBER.keyword": [
"99"
],
"data.buildVariables.BUILD_TAG": [
"jenkinsjenkins-win-configure-99"
],
"data.buildVariables.BUILD_TAG.keyword": [
"jenkins-jenkins-win-configure-99"
],
"data.buildVariables.BUILD_URL": [
"https://.com/jobjenkins/job/win-configure/99/"
],
"data.buildVariables.BUILD_URL.keyword": [
"https://com/jobnkins/job/win-configure/99/"
],
"data.buildVariables.CI": [
"false"
],
"data.buildVariables.CI.keyword": [
"false"
],
"data.buildVariables.CLASSPATH": [
""
],
"data.buildVariables.CLASSPATH.keyword": [
""
],
"data.buildVariables.GCP_PROMOTE": [
"true"
],
"data.buildVariables.GCP_PROMOTE.keyword": [
"true"
],
"data.buildVariables.HUDSON_HOME": [
"/mnt/jenkins"
],
"data.buildVariables.HUDSON_HOME.keyword": [
"/mnt/jenkins"
],
"data.buildVariables.HUDSON_SERVER_COOKIE": [
"f78b065630717635"
],
"data.buildVariables.HUDSON_SERVER_COOKIE.keyword": [
"f78b065630717635"
],
"data.buildVariables.HUDSON_URL": [
"https://.com/"
],
"data.buildVariables.HUDSON_URL.keyword": [
"https://com/"
],
"data.buildVariables.JENKINS_HOME": [
"/mnt/jenkins"
],
"data.buildVariables.JENKINS_HOME.keyword": [
"/mnt/jenkins"
],
"data.buildVariables.JENKINS_SERVER_COOKIE": [
"f78b065630717635"
],
"data.buildVariables.JENKINS_SERVER_COOKIE.keyword": [
"f78b065630717635"
],
"data.buildVariables.JENKINS_URL": [
"https://com/"
],
"data.buildVariables.JENKINS_URL.keyword": [
"https://.com/"
],
"data.buildVariables.JOB_BASE_NAME": [
"win-configure"
],
"data.buildVariables.JOB_BASE_NAME.keyword": [
"win-configure"
],
"data.buildVariables.JOB_DISPLAY_URL": [
"https://job/win-configure/display/redirect"
],
"data.buildVariables.JOB_DISPLAY_URL.keyword": [
"https:///job/win-configure/display/redirect"
],
"data.buildVariables.JOB_NAME": [
"enkins/win-configure"
],
"data.buildVariables.JOB_NAME.keyword": [
"enkins/win-configure"
],
"data.buildVariables.JOB_URL": [
"https://c/job/win-configure/"
],
"data.buildVariables.JOB_URL.keyword": [
"https:///job/win-configure/"
],
"data.buildVariables.PROMOTION_URL": [
"https://gcp.promote.repository.com"
],
"data.buildVariables.PROMOTION_URL.keyword": [
"https://gcp.promote.repository.com"
],
"data.buildVariables.REPOSITORY": [
"repository..com"
],
"data.buildVariables.REPOSITORY.keyword": [
"repositorycom"
],
"data.buildVariables.RUN_ARTIFACTS_DISPLAY_URL": [
"https:///job/win-configure/99/display/redirect?page=artifacts"
],
"data.buildVariables.RUN_ARTIFACTS_DISPLAY_URL.keyword": [
"https://.com/job/-jenkins/job/win-configure/99/display/redirect?page=artifacts"
],
"data.buildVariables.RUN_CHANGES_DISPLAY_URL": [
"https://com/jobenkins/job/win-configure/99/display/redirect?page=changes"
],
"data.buildVariables.RUN_CHANGES_DISPLAY_URL.keyword": [
"https:///job/win-configure/99/display/redirect?page=changes"
],
"data.buildVariables.RUN_DISPLAY_URL": [
"https://com/jobjenkins/job/win-configure/99/display/redirect"
],
"data.buildVariables.RUN_DISPLAY_URL.keyword": [
"https://.com/jobenkins/job/win-configure/99/display/redirect"
],
"data.buildVariables.RUN_TESTS_DISPLAY_URL": [
"https://com/jobjenkins/job/win-configure/99/display/redirect?page=tests"
],
"data.buildVariables.RUN_TESTS_DISPLAY_URL.keyword": [
"https://com/jobnkins/job/win-configure/99/display/redirect?page=tests"
],
"data.buildVariables.RUN_VERACODE": [
"false"
],
"data.buildVariables.RUN_VERACODE.keyword": [
"false"
],
"data.buildVariables.SKIP_SONAR": [
"true"
],
"data.buildVariables.SKIP_SONAR.keyword": [
"true"
],
"data.displayName": [
"#99"
],
"data.displayName.keyword": [
"#99"
],
"data.fullDisplayName": [
"jenkins » win-configure #99"
],
"data.fullDisplayName.keyword": [
"jenkins » win-configure #99"
],
"data.fullProjectName": [
"jenkins/win-configure"
],
"data.fullProjectName.keyword": [
"nkins/win-configure"
],
"data.id": [
"99"
],
"data.id.keyword": [
"99"
],
"data.projectName": [
"win-configure"
],
"data.projectName.keyword": [
"win-configure"
],
"data.rootBuildNum": [
99
],
"data.rootFullProjectName": [
"/win-configure"
],
"data.rootFullProjectName.keyword": [
"/win-configure"
],
"data.rootProjectDisplayName": [
"#99"
],
"data.rootProjectDisplayName.keyword": [
"#99"
],
"data.rootProjectName": [
"win-configure"
],
"data.rootProjectName.keyword": [
"win-configure"
],
"data.url": [
"job/job/win-configure/99/"
],
"data.url.keyword": [
"job//job/win-configure/99/"
],
"message": [
"[2023-08-04T14:01:27.432Z] Started by user",
"[2023-08-04T14:01:27.601Z] [Pipeline] Start of Pipeline",
"[2023-08-04T14:01:27.739Z] [Pipeline] node",
"[2023-08-04T14:01:42.746Z] Still waiting to schedule task",
"[2023-08-04T14:01:42.746Z] ‘agent-s6hd8c’ is offline",
"[2023-08-04T14:04:10.866Z] Running ons-agent-s6hd8c in /tmp/workspace/jenkins/win-configure",
"[2023-08-04T14:04:10.873Z] [Pipeline] {",
"[2023-08-04T14:04:10.920Z] [Pipeline] stage",
"[2023-08-04T14:04:10.926Z] [Pipeline] { (Set Ansible Version)",
"[2023-08-04T14:04:10.971Z] [Pipeline] script",
"[2023-08-04T14:04:10.977Z] [Pipeline] {",
"[2023-08-04T14:04:11.021Z] [Pipeline] httpRequest",
"[2023-08-04T14:04:11.253Z] HttpMethod: GET",
"[2023-08-04T14:04:11.253Z] URL: https://gcp.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=com.&name=-ansible&version=",
"[2023-08-04T14:04:11.341Z] Using authentication: -repository-token",
"[2023-08-04T14:04:11.841Z] Sending request to url: https://.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=com&name=nkins-win-ansible&version=",
"[2023-08-04T14:04:12.299Z] Response Code: HTTP/1.1 200 OK",
"[2023-08-04T14:04:12.345Z] Success: Status code 200 is in the accepted range: 100:399",
"[2023-08-04T14:04:12.364Z] [Pipeline] readJSON",
"[2023-08-04T14:04:12.407Z] [Pipeline] echo",
"[2023-08-04T14:04:12.413Z] all matching versions=[1.0.0-2-g26dc582]",
"[2023-08-04T14:04:12.424Z] [Pipeline] echo",
"[2023-08-04T14:04:12.430Z] selected version=1.0.0-2-g26dc582",
"[2023-08-04T14:04:12.441Z] [Pipeline] sh",
"[2023-08-04T14:04:12.747Z] + gcloud compute project-info add-metadata --metadata -win-ansible-version=1.0.0-2-g26dc582",
"[2023-08-04T14:04:14.129Z] No change requested; skipping update for [sab-dev5697].",
"[2023-08-04T14:04:14.417Z] [Pipeline] sleep",
"[2023-08-04T14:04:14.423Z] Sleeping for 3 sec",
"[2023-08-04T14:04:17.442Z] [Pipeline] logstashSend"
],
"message.keyword": [
"[2023-08-04T14:01:27.432Z] Started by user ",
"[2023-08-04T14:01:27.601Z] [Pipeline] Start of Pipeline",
"[2023-08-04T14:01:27.739Z] [Pipeline] node",
"[2023-08-04T14:01:42.746Z] Still waiting to schedule task",
"[2023-08-04T14:01:42.746Z] ‘-agent-s6hd8c’ is offline",
"[2023-08-04T14:04:10.866Z] Running on -jenkins-agent-s6hd8c in /tmp/workspace//win-configure",
"[2023-08-04T14:04:10.873Z] [Pipeline] {",
"[2023-08-04T14:04:10.920Z] [Pipeline] stage",
"[2023-08-04T14:04:10.926Z] [Pipeline] { (Set Ansible Version)",
"[2023-08-04T14:04:10.971Z] [Pipeline] script",
"[2023-08-04T14:04:10.977Z] [Pipeline] {",
"[2023-08-04T14:04:11.021Z] [Pipeline] httpRequest",
"[2023-08-04T14:04:11.253Z] HttpMethod: GET",
"[2023-08-04T14:04:11.253Z] URL: https://repository.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=comhen&name=win-ansible&version=",
"[2023-08-04T14:04:11.341Z] Using authentication: repository-token",
"[2023-08-04T14:04:11.841Z] Sending request to url: https://gcp.com/service/rest/v1/search/assets?repository=cloud-templates-trusted&group=com.&name=-win-ansible&version=",
"[2023-08-04T14:04:12.299Z] Response Code: HTTP/1.1 200 OK",
"[2023-08-04T14:04:12.345Z] Success: Status code 200 is in the accepted range: 100:399",
"[2023-08-04T14:04:12.364Z] [Pipeline] readJSON",
"[2023-08-04T14:04:12.407Z] [Pipeline] echo",
"[2023-08-04T14:04:12.413Z] all matching versions=[1.0.0-2-g26dc582]",
"[2023-08-04T14:04:12.424Z] [Pipeline] echo",
"[2023-08-04T14:04:12.430Z] selected version=1.0.0-2-g26dc582",
"[2023-08-04T14:04:12.441Z] [Pipeline] sh",
"[2023-08-04T14:04:12.747Z] + gcloud compute project-info add-metadata --metadata -win-ansible-version=1.0.0-2-g26dc582",
"[2023-08-04T14:04:14.129Z] No change requested; skipping update for [ore-5697].",
"[2023-08-04T14:04:14.417Z] [Pipeline] sleep",
"[2023-08-04T14:04:14.423Z] Sleeping for 3 sec",
"[2023-08-04T14:04:17.442Z] [Pipeline] logstashSend"
],
"source": [
"jenkins"
],
"source_host": [
"https:/.com/"
],
"source_host.keyword": [
"https://.com/"
],
"source.keyword": [
"jenkins"
],
"_id": "CirdwIkBDKSsBQJEBEiy",
"_index": "jenkins-log",
"_score": null
}
```
---
### Author:  [@Jenkins-Jobs](https://discuss.elastic.co/u/Jenkins-Jobs)
#### Post date: [August 21, 2023, 4:04pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/8 "2023-08-21T16:04:30Z")
my response was hidden for a little while, but this is the contect with references to infrastructure removed
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 18, 2023, 4:05pm UTC](https://discuss.elastic.co/t/cannot-change-log-format-with-pipeline-config-file-pipeline-config-file-is-not-getting-read/340081/9 "2023-09-18T16:05:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.