DataHubScheduler - Details for Query 5958

Submitted Time: 2026/10/16 08:20:48
Duration: 4 s
Succeeded Jobs: 9099 9105 9110

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 14351.0: task 31937)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 16.0 B (0.0 B, 16.0 B, 16.0 B (stage 14351.0: task 31937)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 59.0 B (0.0 B, 59.0 B, 59.0 B (stage 14351.0: task 31937))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 0 ms"; 5 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 6 [labelType="html" label="InMemoryTableScan number of output rows: 146"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n37 ms (0 ms, 37 ms, 37 ms (stage 14343.0: task 31931))"; 8 [labelType="html" label="Scan JDBCRelation((SELECT * FROM dbo.fact_scm_supplier_compliance_pip_documents WHERE bigid BETWEEN 1 AND 146) AS page) [numPartitions=1] number of output rows: 146"]; } 2->0; 3->2; 5->3; 6->5; 8->6; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (2)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=369389]

HashAggregate(keys=[], functions=[partial_count(1)])

WholeStageCodegen (1)

InMemoryTableScan

Scan JDBCRelation((SELECT * FROM dbo.fact_scm_supplier_compliance_pip_documents WHERE bigid BETWEEN 1 AND 146) AS page) [numPartitions=1] [brand_context_id#10539180,supplier_id#10539181,seller_metadata_name#10539182,seller_metadata_uuid#10539183,country#10539184,delivery_date#10539185,expiry_date#10539186,create_date#10539187,po_metadata_id#10539188,po_metadata_name#10539189,po_quantity#10539190,po_tracing_metadata_id#10539191,po_tracing_metadata_uuid#10539192,product_metadata_id#10539193,product_metadata_name#10539194,product_metadata_uuid#10539195,scm_config_name#10539196,status#10539197,uuid#10539198,shipment_id#10539199,shipped_quantity#10539200,supply_chain_tree_response_uuid#10539201,launch_config_ref#10539202,Supplier_Name#10539203,... 6 more fields] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,seller_metadata_name:string,seller_metadata_uui...

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (12)
+- == Final Plan ==
   * HashAggregate (8)
   +- ShuffleQueryStage (7), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (6)
         +- * HashAggregate (5)
            +- TableCacheQueryStage (4), Statistics(sizeInBytes=63.8 KiB, rowCount=146)
               +- InMemoryTableScan (1)
                     +- InMemoryRelation (2)
                           +- * Scan JDBCRelation((SELECT * FROM dbo.fact_scm_supplier_compliance_pip_documents WHERE bigid BETWEEN 1 AND 146) AS page) [numPartitions=1]  (3)
+- == Initial Plan ==
   HashAggregate (11)
   +- Exchange (10)
      +- HashAggregate (9)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- * Scan JDBCRelation((SELECT * FROM dbo.fact_scm_supplier_compliance_pip_documents WHERE bigid BETWEEN 1 AND 146) AS page) [numPartitions=1]  (3)


(1) InMemoryTableScan
Output: []

(2) InMemoryRelation
Arguments: [brand_context_id#10539180, supplier_id#10539181, seller_metadata_name#10539182, seller_metadata_uuid#10539183, country#10539184, delivery_date#10539185, expiry_date#10539186, create_date#10539187, po_metadata_id#10539188, po_metadata_name#10539189, po_quantity#10539190, po_tracing_metadata_id#10539191, po_tracing_metadata_uuid#10539192, product_metadata_id#10539193, product_metadata_name#10539194, product_metadata_uuid#10539195, scm_config_name#10539196, status#10539197, uuid#10539198, shipment_id#10539199, shipped_quantity#10539200, supply_chain_tree_response_uuid#10539201, launch_config_ref#10539202, Supplier_Name#10539203, ... 6 more fields], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@7795a5ad,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Scan JDBCRelation((SELECT * FROM dbo.fact_scm_supplier_compliance_pip_documents WHERE bigid BETWEEN 1 AND 146) AS page) [numPartitions=1] [brand_context_id#10539180,supplier_id#10539181,seller_metadata_name#10539182,seller_metadata_uuid#10539183,country#10539184,delivery_date#10539185,expiry_date#10539186,create_date#10539187,po_metadata_id#10539188,po_metadata_name#10539189,po_quantity#10539190,po_tracing_metadata_id#10539191,po_tracing_metadata_uuid#10539192,product_metadata_id#10539193,product_metadata_name#10539194,product_metadata_uuid#10539195,scm_config_name#10539196,status#10539197,uuid#10539198,shipment_id#10539199,shipped_quantity#10539200,supply_chain_tree_response_uuid#10539201,launch_config_ref#10539202,Supplier_Name#10539203,... 6 more fields] PushedFilters: [], ReadSchema: struct<brand_context_id:string,supplier_id:string,seller_metadata_name:string,seller_metadata_uui...
,None)

(3) Scan JDBCRelation((SELECT * FROM dbo.fact_scm_supplier_compliance_pip_documents WHERE bigid BETWEEN 1 AND 146) AS page) [numPartitions=1]  [codegen id : 1]
Output [30]: [brand_context_id#10539180, supplier_id#10539181, seller_metadata_name#10539182, seller_metadata_uuid#10539183, country#10539184, delivery_date#10539185, expiry_date#10539186, create_date#10539187, po_metadata_id#10539188, po_metadata_name#10539189, po_quantity#10539190, po_tracing_metadata_id#10539191, po_tracing_metadata_uuid#10539192, product_metadata_id#10539193, product_metadata_name#10539194, product_metadata_uuid#10539195, scm_config_name#10539196, status#10539197, uuid#10539198, shipment_id#10539199, shipped_quantity#10539200, supply_chain_tree_response_uuid#10539201, launch_config_ref#10539202, Supplier_Name#10539203, Product_Name#10539204, tier#10539205, Requested_Count#10539206, Missing_Count#10539207, Uploaded_Count#10539208, bigid#10539209L]
ReadSchema: struct<brand_context_id:string,supplier_id:string,seller_metadata_name:string,seller_metadata_uuid:string,country:string,delivery_date:date,expiry_date:date,create_date:date,po_metadata_id:string,po_metadata_name:string,po_quantity:double,po_tracing_metadata_id:string,po_tracing_metadata_uuid:string,product_metadata_id:string,product_metadata_name:string,product_metadata_uuid:string,scm_config_name:string,status:string,uuid:string,shipment_id:string,shipped_quantity:double,supply_chain_tree_response_uuid:string,launch_config_ref:string,Supplier_Name:string,Product_Name:string,tier:smallint,Requested_Count:int,Missing_Count:int,Uploaded_Count:int,bigid:bigint>

(4) TableCacheQueryStage
Output: []
Arguments: 0

(5) HashAggregate [codegen id : 1]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#10539873L]
Results [1]: [count#10539874L]

(6) Exchange
Input [1]: [count#10539874L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=369389]

(7) ShuffleQueryStage
Output [1]: [count#10539874L]
Arguments: 1

(8) HashAggregate [codegen id : 2]
Input [1]: [count#10539874L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#10539420L]
Results [1]: [count(1)#10539420L AS count#10539421L]

(9) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#10539873L]
Results [1]: [count#10539874L]

(10) Exchange
Input [1]: [count#10539874L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=369292]

(11) HashAggregate
Input [1]: [count#10539874L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#10539420L]
Results [1]: [count(1)#10539420L AS count#10539421L]

(12) AdaptiveSparkPlan
Output [1]: [count#10539421L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.shuffle.partitions	10