# Parse json in pipeline **URL:** \ **Category:** Elastic Agent\ **Created:** [October 24, 2023, 7:22am UTC](https://discuss.elastic.co/t/parse-json-in-pipeline/345612 "2023-10-24T07:22:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![volkerfrank](https://avatars.discourse-cdn.com/v4/letter/v/e36b37/32.png) [@volkerfrank](https://discuss.elastic.co/u/volkerfrank)\ **Post date:** [October 24, 2023, 7:22am UTC](https://discuss.elastic.co/t/parse-json-in-pipeline/345612/1 "2023-10-24T07:22:25Z")
When I ingest json logs over a pipeline with a json parser, I get a document\_parsing\_exception error: ```auto (status = 400): { "type": "document_parsing_exception", "reason": "[1:15872] failed to parse field [gitlab.params.value] of type [keyword] in document with id 'LcFzYIsBEzyvtGgrvrS5'. Preview of field's value: '{variables=[FILTERED], query=query pathLastCommit($projectPath: ID!, $path: String, $ref: String!, $refType: RefType) {\n project(fullPath: $projectPath) {\n __typename\n id\n repository {\n__ typename\n paginatedTree(path: $path, ref: $ref, refType: $refType) {\n __typename\n nodes {\n__ typename\n lastCommit {\n __typename\n id\n sha\n title\n titleHtml\n descriptionHtml\n message\n webPath\n authoredDate\n authorName\n authorGravatar\n author {\n__ typename\n id\n name\n avatarUrl\n webPath\n }\n signature {\n __typename\n ... on GpgSignature {\n gpgKeyPrimaryKeyid\n verificationStatus\n__ typename\n }\n ... on X509Signature {\n verificationStatus\n x509Certificate {\n id\n subject\n subjectKeyIdentifier\n x509Issuer {\n id\n subject\n subjectKeyIdentifier\n __typename\n }\n__ typename\n }\n __typename\n }\n ... on SshSignature {\n verificationStatus\n keyFingerprintSha256\n__ typename\n }\n }\n pipelines(ref: $ref, first: 1) {\n __typename\n edges {\n__ typename\n node {\n __typename\n id\n detailedStatus {\n__ typename\n id\n detailsPath\n icon\n tooltip\n text\n group\n }\n }\n }\n }\n }\n }\n }\n }\n }\n}\n, operationName=pathLastCommit}'", "caused_by": { "type": "illegal_state_exception", "reason": "Can't get text on a START_OBJECT at 1:13668" } } ``` I write the parsed fields to the target field gitlab. The problem is the content of the parsed field params.value. The content "[FILTERED]" is interpreted as array I think. But where I simulate this pipeline with the same document, the json is parsed without error. The ignore\_failure flag is true, but this don't help. Any ideas to solve this? Here is my simulate request: ```auto "pipeline": { "description": "parse json", "processors": [ { "json": { "field": "message", "target_field": "gitlab", "ignore_failure": false, "strict_json_parsing": false } } ] }, "docs": [ { "_source": { "agent": { "ephemeral_id": "d8c092e5-5b6d-48d2-a0c4-97e770e0fb65", "id": "6fb7162d-8150-4969-8a32-f1526e6e0db7", "name": "progit2-1", "type": "filebeat", "version": "8.10.2" }, "container": { "id": "gitlab-rails" }, "data_stream": { "dataset": "gitlab", "namespace": "pws", "type": "logs" }, "ecs": { "version": "8.0.0" }, "elastic_agent": { "id": "6fb7162d-8150-4969-8a32-f1526e6e0db7", "snapshot": false, "version": "8.10.2" }, "event": { "dataset": "gitlab" }, "host": { "architecture": "x86_64", "containerized": false, "hostname": "progit2-1", "id": "23df0f2fa04b415182d58c2dbe5798b8", "ip": [ "10.210.1.112", "fe80::250:56ff:fe97:4aaa", "172.17.0.193", "fe80::42:13ff:feee:4587", "172.17.0.1", "fe80::8075:14ff:fef6:707b" ], "mac": [ "00-50-56-97-4A-AA", "02-42-13-EE-45-87", "02-42-56-AE-7B-1E", "82-75-14-F6-70-7B" ], "name": "progit2-1", "os": { "codename": "bullseye", "family": "debian", "kernel": "5.10.0-26-amd64", "name": "Debian GNU/Linux", "platform": "debian", "type": "linux", "version": "11 (bullseye)" } }, "input": { "type": "log" }, "log": { "file": { "path": "/var/lib/gitlab/logs/gitlab-rails/production_json.log" }, "offset": 7082597 }, "message": "{\"method\":\"POST\",\"path\":\"/api/graphql\",\"format\":\"*/*\",\"controller\":\"GraphqlController\",\"action\":\"execute\",\"status\":200,\"time\":\"2023-10-24T06:51:18.708Z\",\"params\":[{\"key\":\"operationName\",\"value\":\"pathLastCommit\"},{\"key\":\"variables\",\"value\":\"[FILTERED]\"},{\"key\":\"query\",\"value\":\"query pathLastCommit($projectPath: ID!, $path: String, $ref: String!, $refType: RefType) {\\n project(fullPath: $projectPath) {\\n __typename\\n id\\n repository {\\n__ typename\\n paginatedTree(path: $path, ref: $ref, refType: $refType) {\\n __typename\\n nodes {\\n__ typename\\n lastCommit {\\n __typename\\n id\\n sha\\n title\\n titleHtml\\n descriptionHtml\\n message\\n webPath\\n authoredDate\\n authorName\\n authorGravatar\\n author {\\n__ typename\\n id\\n name\\n avatarUrl\\n webPath\\n }\\n signature {\\n __typename\\n ... on GpgSignature {\\n gpgKeyPrimaryKeyid\\n verificationStatus\\n__ typename\\n }\\n ... on X509Signature {\\n verificationStatus\\n x509Certificate {\\n id\\n subject\\n subjectKeyIdentifier\\n x509Issuer {\\n id\\n subject\\n subjectKeyIdentifier\\n __typename\\n }\\n__ typename\\n }\\n __typename\\n }\\n ... on SshSignature {\\n verificationStatus\\n keyFingerprintSha256\\n__ typename\\n }\\n }\\n pipelines(ref: $ref, first: 1) {\\n __typename\\n edges {\\n__ typename\\n node {\\n __typename\\n id\\n detailedStatus {\\n__ typename\\n id\\n detailsPath\\n icon\\n tooltip\\n text\\n group\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n}\\n\"},{\"key\":\"graphql\",\"value\":{\"operationName\":\"pathLastCommit\",\"variables\":\"[FILTERED]\",\"query\":\"query pathLastCommit($projectPath: ID!, $path: String, $ref: String!, $refType: RefType) {\\n project(fullPath: $projectPath) {\\n __typename\\n id\\n repository {\\n__ typename\\n paginatedTree(path: $path, ref: $ref, refType: $refType) {\\n __typename\\n nodes {\\n__ typename\\n lastCommit {\\n __typename\\n id\\n sha\\n title\\n titleHtml\\n descriptionHtml\\n message\\n webPath\\n authoredDate\\n authorName\\n authorGravatar\\n author {\\n__ typename\\n id\\n name\\n avatarUrl\\n webPath\\n }\\n signature {\\n __typename\\n ... on GpgSignature {\\n gpgKeyPrimaryKeyid\\n verificationStatus\\n__ typename\\n }\\n ... on X509Signature {\\n verificationStatus\\n x509Certificate {\\n id\\n subject\\n subjectKeyIdentifier\\n x509Issuer {\\n id\\n subject\\n subjectKeyIdentifier\\n __typename\\n }\\n__ typename\\n }\\n __typename\\n }\\n ... on SshSignature {\\n verificationStatus\\n keyFingerprintSha256\\n__ typename\\n }\\n }\\n pipelines(ref: $ref, first: 1) {\\n __typename\\n edges {\\n__ typename\\n node {\\n __typename\\n id\\n detailedStatus {\\n__ typename\\n id\\n detailsPath\\n icon\\n tooltip\\n text\\n group\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n }\\n}\\n\"}}],\"correlation_id\":\"01HDG77E63G7QGRWH8Y70Z9G9X\",\"meta.caller_id\":\"GraphqlController#execute\",\"meta.remote_ip\":\"10.212.134.35\",\"meta.feature_category\":\"groups_and_projects\",\"meta.user\":\"allgaier\",\"meta.user_id\":18,\"meta.client_id\":\"user/18\",\"graphql\":[{\"depth\":10,\"complexity\":77,\"used_fields\":[\"Project. __typename\",\"Project.id\",\"Repository.__ typename\",\"TreeConnection. __typename\",\"Tree.__ typename\",\"Commit. __typename\",\"Commit.id\",\"Commit.sha\",\"Commit.title\",\"Commit.titleHtml\",\"Commit.descriptionHtml\",\"Commit.message\",\"Commit.webPath\",\"Commit.authoredDate\",\"Commit.authorName\",\"Commit.authorGravatar\",\"UserCore.__ typename\",\"UserCore.id\",\"UserCore.name\",\"UserCore.avatarUrl\",\"UserCore.webPath\",\"Commit.author\",\"CommitSignature. __typename\",\"GpgSignature.gpgKeyPrimaryKeyid\",\"GpgSignature.verificationStatus\",\"GpgSignature.__ typename\",\"X509Signature.verificationStatus\",\"X509Certificate.id\",\"X509Certificate.subject\",\"X509Certificate.subjectKeyIdentifier\",\"X509Issuer.id\",\"X509Issuer.subject\",\"X509Issuer.subjectKeyIdentifier\",\"X509Issuer. __typename\",\"X509Certificate.x509Issuer\",\"X509Certificate.__ typename\",\"X509Signature.x509Certificate\",\"X509Signature. __typename\",\"SshSignature.verificationStatus\",\"SshSignature.keyFingerprintSha256\",\"SshSignature.__ typename\",\"Commit.signature\",\"PipelineConnection. __typename\",\"PipelineEdge.__ typename\",\"Pipeline. __typename\",\"Pipeline.id\",\"DetailedStatus.__ typename\",\"DetailedStatus.id\",\"DetailedStatus.detailsPath\",\"DetailedStatus.icon\",\"DetailedStatus.tooltip\",\"DetailedStatus.text\",\"DetailedStatus.group\",\"Pipeline.detailedStatus\",\"PipelineEdge.node\",\"PipelineConnection.edges\",\"Commit.pipelines\",\"Tree.lastCommit\",\"TreeConnection.nodes\",\"Repository.paginatedTree\",\"Project.repository\",\"Query.project\"],\"used_deprecated_fields\":[],\"variables\":\"{\\\"projectPath\\\"=\u003e\\\"infrastructure-as-code/iac-rabbitmq\\\", \\\"ref\\\"=\u003e\\\"master\\\", \\\"refType\\\"=\u003e\\\"HEADS\\\", \\\"path\\\"=\u003e\\\"scripts/deploy.sh\\\"}\",\"operation_name\":\"pathLastCommit\"}],\"remote_ip\":\"10.212.134.35\",\"user_id\":18,\"username\":\"allgaier\",\"ua\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0\",\"queue_duration_s\":0.006815,\"request_urgency\":\"low\",\"target_duration_s\":5,\"gitaly_calls\":2,\"gitaly_duration_s\":0.01107,\"redis_calls\":8,\"redis_allowed_cross_slot_calls\":1,\"redis_duration_s\":0.002509,\"redis_read_bytes\":491,\"redis_write_bytes\":969,\"redis_cache_calls\":1,\"redis_cache_duration_s\":0.001522,\"redis_cache_read_bytes\":109,\"redis_cache_write_bytes\":131,\"redis_repository_cache_calls\":2,\"redis_repository_cache_duration_s\":0.000288,\"redis_repository_cache_read_bytes\":202,\"redis_repository_cache_write_bytes\":137,\"redis_sessions_calls\":3,\"redis_sessions_allowed_cross_slot_calls\":1,\"redis_sessions_duration_s\":0.000518,\"redis_sessions_read_bytes\":180,\"redis_sessions_write_bytes\":595,\"redis_shared_state_calls\":2,\"redis_shared_state_duration_s\":0.000181,\"redis_shared_state_write_bytes\":106,\"db_count\":12,\"db_write_count\":0,\"db_cached_count\":0,\"db_replica_count\":0,\"db_primary_count\":12,\"db_main_count\":10,\"db_ci_count\":2,\"db_main_replica_count\":0,\"db_ci_replica_count\":0,\"db_replica_cached_count\":0,\"db_primary_cached_count\":0,\"db_main_cached_count\":0,\"db_ci_cached_count\":0,\"db_main_replica_cached_count\":0,\"db_ci_replica_cached_count\":0,\"db_replica_wal_count\":0,\"db_primary_wal_count\":0,\"db_main_wal_count\":0,\"db_ci_wal_count\":0,\"db_main_replica_wal_count\":0,\"db_ci_replica_wal_count\":0,\"db_replica_wal_cached_count\":0,\"db_primary_wal_cached_count\":0,\"db_main_wal_cached_count\":0,\"db_ci_wal_cached_count\":0,\"db_main_replica_wal_cached_count\":0,\"db_ci_replica_wal_cached_count\":0,\"db_replica_duration_s\":0.0,\"db_primary_duration_s\":0.023,\"db_main_duration_s\":0.011,\"db_ci_duration_s\":0.012,\"db_main_replica_duration_s\":0.0,\"db_ci_replica_duration_s\":0.0,\"cpu_s\":0.062784,\"mem_objects\":32232,\"mem_bytes\":2056312,\"mem_mallocs\":10556,\"mem_total_bytes\":3345592,\"pid\":338000,\"worker_id\":\"puma_1\",\"rate_limiting_gates\":[],\"db_duration_s\":0.02142,\"view_duration_s\":0.00034,\"duration_s\":0.10015}" } } ] } ``` ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [October 24, 2023, 10:15am UTC](https://discuss.elastic.co/t/parse-json-in-pipeline/345612/2 "2023-10-24T10:15:20Z")
Hi @volkerfrank, Welcome to the community! Can you share the configuration for your json processor as well? ---
**Author:** ![volkerfrank](https://avatars.discourse-cdn.com/v4/letter/v/e36b37/32.png) [@volkerfrank](https://discuss.elastic.co/u/volkerfrank)\ **Post date:** [October 24, 2023, 1:28pm UTC](https://discuss.elastic.co/t/parse-json-in-pipeline/345612/3 "2023-10-24T13:28:36Z")
This is the configuration of the json processor: ```auto "processors": [ { "json": { "field": "message", "target_field": "gitlab", "ignore_failure": true, "strict_json_parsing": false } } ] ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 21, 2023, 1:28pm UTC](https://discuss.elastic.co/t/parse-json-in-pipeline/345612/4 "2023-11-21T13:28:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.