DataHubScheduler - Details for Query 6037

Submitted Time: 2026/10/16 08:21:07
Duration: 0.3 s
Succeeded Jobs: 9216

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" DeserializeToObject "]; 2 [labelType="html" label="InMemoryTableScan number of output rows: 1,925"]; subgraph cluster3 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 0 ms"; 4 [labelType="html" label="Scan JDBCRelation((SELECT * FROM dbo.fact_spa_audit_violation_summary_nc WHERE bigid BETWEEN 1 AND 1925) AS page) [numPartitions=1] number of output rows: 0"]; } 1->0; 2->1; 4->2; }

AdaptiveSparkPlan isFinalPlan=true

DeserializeToObject createexternalrow(brandid#10588235.toString, AuditRequestID#10588236.toString, TemplateID#10588237.toString, AuditType#10588238.toString, FacilityID#10588239.toString, sup_type#10588240.toString, staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$, ObjectType(class java.sql.Date), toJavaDate, audit_date#10588241, true, false, true), TitleName#10588242.toString, SubtitleName#10588243.toString, HeadingLabel#10588244.toString, audit_checklist_severity#10588245.toString, audit_checklist_updated_severity#10588246.toString, audit_checklist_assessor_response#10588247.toString, EffectiveSeverity#10588248.toString, StructField(brandid,StringType,true), StructField(AuditRequestID,StringType,true), StructField(TemplateID,StringType,true), StructField(AuditType,StringType,true), StructField(FacilityID,StringType,true), StructField(sup_type,StringType,true), StructField(audit_date,DateType,true), StructField(TitleName,StringType,true), StructField(SubtitleName,StringType,true), StructField(HeadingLabel,StringType,true), ... 4 more fields), obj#10591134: org.apache.spark.sql.Row

InMemoryTableScan [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248]

Scan JDBCRelation((SELECT * FROM dbo.fact_spa_audit_violation_summary_nc WHERE bigid BETWEEN 1 AND 1925) AS page) [numPartitions=1] [brandid#10588235,AuditRequestID#10588236,TemplateID#10588237,AuditType#10588238,FacilityID#10588239,sup_type#10588240,audit_date#10588241,TitleName#10588242,SubtitleName#10588243,HeadingLabel#10588244,audit_checklist_severity#10588245,audit_checklist_updated_severity#10588246,audit_checklist_assessor_response#10588247,EffectiveSeverity#10588248,bigid#10588249L] PushedFilters: [], ReadSchema: struct<brandid:string,AuditRequestID:string,TemplateID:string,AuditType:string,FacilityID:string,...

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (7)
+- == Final Plan ==
   DeserializeToObject (5)
   +- TableCacheQueryStage (4), Statistics(sizeInBytes=495.4 KiB, rowCount=1.93E+3)
      +- InMemoryTableScan (1)
            +- InMemoryRelation (2)
                  +- * Scan JDBCRelation((SELECT * FROM dbo.fact_spa_audit_violation_summary_nc WHERE bigid BETWEEN 1 AND 1925) AS page) [numPartitions=1]  (3)
+- == Initial Plan ==
   DeserializeToObject (6)
   +- InMemoryTableScan (1)
         +- InMemoryRelation (2)
               +- * Scan JDBCRelation((SELECT * FROM dbo.fact_spa_audit_violation_summary_nc WHERE bigid BETWEEN 1 AND 1925) AS page) [numPartitions=1]  (3)


(1) InMemoryTableScan
Output [14]: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248]
Arguments: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248]

(2) InMemoryRelation
Arguments: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248, bigid#10588249L], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@7795a5ad,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Scan JDBCRelation((SELECT * FROM dbo.fact_spa_audit_violation_summary_nc WHERE bigid BETWEEN 1 AND 1925) AS page) [numPartitions=1] [brandid#10588235,AuditRequestID#10588236,TemplateID#10588237,AuditType#10588238,FacilityID#10588239,sup_type#10588240,audit_date#10588241,TitleName#10588242,SubtitleName#10588243,HeadingLabel#10588244,audit_checklist_severity#10588245,audit_checklist_updated_severity#10588246,audit_checklist_assessor_response#10588247,EffectiveSeverity#10588248,bigid#10588249L] PushedFilters: [], ReadSchema: struct<brandid:string,AuditRequestID:string,TemplateID:string,AuditType:string,FacilityID:string,...
,None)

(3) Scan JDBCRelation((SELECT * FROM dbo.fact_spa_audit_violation_summary_nc WHERE bigid BETWEEN 1 AND 1925) AS page) [numPartitions=1]  [codegen id : 1]
Output [15]: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248, bigid#10588249L]
ReadSchema: struct<brandid:string,AuditRequestID:string,TemplateID:string,AuditType:string,FacilityID:string,sup_type:string,audit_date:date,TitleName:string,SubtitleName:string,HeadingLabel:string,audit_checklist_severity:string,audit_checklist_updated_severity:string,audit_checklist_assessor_response:string,EffectiveSeverity:string,bigid:bigint>

(4) TableCacheQueryStage
Output [14]: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248]
Arguments: 0

(5) DeserializeToObject
Input [14]: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248]
Arguments: createexternalrow(brandid#10588235.toString, AuditRequestID#10588236.toString, TemplateID#10588237.toString, AuditType#10588238.toString, FacilityID#10588239.toString, sup_type#10588240.toString, staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$, ObjectType(class java.sql.Date), toJavaDate, audit_date#10588241, true, false, true), TitleName#10588242.toString, SubtitleName#10588243.toString, HeadingLabel#10588244.toString, audit_checklist_severity#10588245.toString, audit_checklist_updated_severity#10588246.toString, audit_checklist_assessor_response#10588247.toString, EffectiveSeverity#10588248.toString, StructField(brandid,StringType,true), StructField(AuditRequestID,StringType,true), StructField(TemplateID,StringType,true), StructField(AuditType,StringType,true), StructField(FacilityID,StringType,true), StructField(sup_type,StringType,true), StructField(audit_date,DateType,true), StructField(TitleName,StringType,true), StructField(SubtitleName,StringType,true), StructField(HeadingLabel,StringType,true), ... 4 more fields), obj#10591134: org.apache.spark.sql.Row

(6) DeserializeToObject
Input [14]: [brandid#10588235, AuditRequestID#10588236, TemplateID#10588237, AuditType#10588238, FacilityID#10588239, sup_type#10588240, audit_date#10588241, TitleName#10588242, SubtitleName#10588243, HeadingLabel#10588244, audit_checklist_severity#10588245, audit_checklist_updated_severity#10588246, audit_checklist_assessor_response#10588247, EffectiveSeverity#10588248]
Arguments: createexternalrow(brandid#10588235.toString, AuditRequestID#10588236.toString, TemplateID#10588237.toString, AuditType#10588238.toString, FacilityID#10588239.toString, sup_type#10588240.toString, staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$, ObjectType(class java.sql.Date), toJavaDate, audit_date#10588241, true, false, true), TitleName#10588242.toString, SubtitleName#10588243.toString, HeadingLabel#10588244.toString, audit_checklist_severity#10588245.toString, audit_checklist_updated_severity#10588246.toString, audit_checklist_assessor_response#10588247.toString, EffectiveSeverity#10588248.toString, StructField(brandid,StringType,true), StructField(AuditRequestID,StringType,true), StructField(TemplateID,StringType,true), StructField(AuditType,StringType,true), StructField(FacilityID,StringType,true), StructField(sup_type,StringType,true), StructField(audit_date,DateType,true), StructField(TitleName,StringType,true), StructField(SubtitleName,StringType,true), StructField(HeadingLabel,StringType,true), ... 4 more fields), obj#10591134: org.apache.spark.sql.Row

(7) AdaptiveSparkPlan
Output [1]: [obj#10591134]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.shuffle.partitions	10