feat: Add support for Spark-compatible explode_outer function for arrays by athlcode · Pull Request #22100 · apache/datafusion

athlcode · 2026-05-10T18:22:36Z

Which issue does this PR close?

Closes Add support for Spark-compatible explode_outer function for arrays #19053.

Rationale for this change

DataFusion's 'unnest' had no way to express Spark 'explode_outer' semantics, empty input lists were silently dropped, even with 'preserve_nulls' = true.

What changes are included in this PR?

Adds a third unnest behavior that produces a NULL row for empty input lists, by replacing UnnestOptions.preserve_nulls: bool with a NullHandling { Drop, Preserve, PreserveAndExpandEmpty } enum. The with_preserve_nulls(bool) builder is kept as a backward-compat shim.

Are these changes tested?

Yes, new unit test for the empty-list case, extended longest-length and DataFrame unnest_column_nulls tests, and existing proto round-trip coverage.

Are there any user-facing changes?

The preserve_nulls field on UnnestOptions is renamed to null_handling. The with_preserve_nulls(bool) builder is preserved, so most callers are unaffected. Add the api change label for the field rename.

github-actions · 2026-05-10T18:32:04Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [  82.525s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.029s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [  81.078s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.029s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.741s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 166.215s] datafusion
    Building datafusion-common v54.0.0 (current)
       Built [  25.048s] (current)
     Parsing datafusion-common v54.0.0 (current)
      Parsed [   0.051s] (current)
    Building datafusion-common v54.0.0 (baseline)
       Built [  25.248s] (baseline)
     Parsing datafusion-common v54.0.0 (baseline)
      Parsed [   0.050s] (baseline)
    Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.819s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field UnnestOptions.null_handling in /home/runner/work/datafusion/datafusion/datafusion/common/src/unnest.rs:104

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field preserve_nulls of struct UnnestOptions, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/65953896f9aebe2965f42e77e42cf6e7d62653ff/datafusion/common/src/unnest.rs:72

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  52.363s] datafusion-common
    Building datafusion-expr v54.0.0 (current)
       Built [  20.179s] (current)
     Parsing datafusion-expr v54.0.0 (current)
      Parsed [   0.060s] (current)
    Building datafusion-expr v54.0.0 (baseline)
       Built [  20.486s] (baseline)
     Parsing datafusion-expr v54.0.0 (baseline)
      Parsed [   0.062s] (baseline)
    Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.579s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field Unnest.outer in /home/runner/work/datafusion/datafusion/datafusion/expr/src/expr.rs:684

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  43.384s] datafusion-expr
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  27.939s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.110s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  27.942s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.111s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.737s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  58.164s] datafusion-physical-plan
    Building datafusion-proto v54.0.0 (current)
       Built [  45.181s] (current)
     Parsing datafusion-proto v54.0.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-proto v54.0.0 (baseline)
       Built [  45.131s] (baseline)
     Parsing datafusion-proto v54.0.0 (baseline)
      Parsed [   0.016s] (baseline)
    Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.296s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  92.207s] datafusion-proto
    Building datafusion-proto-models v54.0.0 (current)
       Built [  18.889s] (current)
     Parsing datafusion-proto-models v54.0.0 (current)
      Parsed [   0.103s] (current)
    Building datafusion-proto-models v54.0.0 (baseline)
       Built [  18.508s] (baseline)
     Parsing datafusion-proto-models v54.0.0 (baseline)
      Parsed [   0.105s] (baseline)
    Checking datafusion-proto-models v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   2.090s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field Unnest.outer in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1006
  field Unnest.outer in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1006
  field UnnestOptions.null_handling in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:671
  field UnnestOptions.null_handling in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:671

--- failure struct_pub_field_missing: pub struct's pub field removed or renamed ---

Description:
A publicly-visible struct has at least one public field that is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/struct_pub_field_missing.ron

Failed in:
  field preserve_nulls of struct UnnestOptions, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/65953896f9aebe2965f42e77e42cf6e7d62653ff/datafusion/proto-models/src/generated/prost.rs:671
  field preserve_nulls of struct UnnestOptions, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/65953896f9aebe2965f42e77e42cf6e7d62653ff/datafusion/proto-models/src/generated/prost.rs:671

     Summary semver requires new major version: 2 major and 0 minor checks failed
    Finished [  41.722s] datafusion-proto-models
    Building datafusion-sql v54.0.0 (current)
       Built [  31.323s] (current)
     Parsing datafusion-sql v54.0.0 (current)
      Parsed [   0.025s] (current)
    Building datafusion-sql v54.0.0 (baseline)
       Built [  31.576s] (baseline)
     Parsing datafusion-sql v54.0.0 (baseline)
      Parsed [   0.026s] (baseline)
    Checking datafusion-sql v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.293s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  64.575s] datafusion-sql
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 141.165s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 141.493s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.098s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 285.394s] datafusion-sqllogictest

…/explode_outer

comphead

Thanks @athlcode explode_outer is not like other builtin function so it has to be in the core.

Please add tests, more tests the better, including mix with nulls, nested combinations with explode and explode_outer and supported datatypes

…/explode_outer

…han7/datafusion into feat/explode_outer

athlcode · 2026-05-13T17:36:17Z

Thanks @athlcode explode_outer is not like other builtin function so it has to be in the core.

Please add tests, more tests the better, including mix with nulls, nested combinations with explode and explode_outer and supported datatypes

thank you, added more test cases

…/explode_outer

comphead · 2026-05-14T15:04:21Z

Thanks @athlcode my bad, I just realized I haven't clarified tests needs to be added to slt file, lets remove rust tests unless they are not reproducible with slt. Ideally to create a separate .slt file for explode_outer to localize tests

athlcode · 2026-05-14T15:16:57Z

Thanks @athlcode my bad, I just realized I haven't clarified tests needs to be added to slt file, lets remove rust tests unless they are not reproducible with slt. Ideally to create a separate .slt file for explode_outer to localize tests

Thanks @comphead for the clarification. None of the test cases are currently reproducible via SLT because there's no SQL surface for NullHandling::PreserveAndExpandEmpty. To enable SLT coverage we need to add SQL recognition of explode_outer(col) (and probably explode(col) for symmetry) and map them to UNNEST with the right NullHandling variant.

comphead · 2026-05-15T23:51:19Z

we would still need to have .slt tests in datafusion-spark crate to test explode_outer works properly with SQL dialect

…/explode_outer

athlcode · 2026-05-18T18:00:30Z

we would still need to have .slt tests in datafusion-spark crate to test explode_outer works properly with SQL dialect

Added slt tests at datafusion/sqllogictest/test_files/spark/generator/explode_outer.slt, exercises explode_outer via the Spark SQL dialect path

kazuyukitanimura · 2026-05-26T22:01:18Z

Thanks @athlcode would you mind resolving the conflicts?

@comphead any further comments?

…/explode_outer

athlcode · 2026-05-27T15:11:20Z

Thanks @athlcode would you mind resolving the conflicts?

@comphead any further comments?

resolved the conflicts, thank you

comphead · 2026-05-27T23:18:28Z

+            })?;
+        }
+    }
+    if saw_outer && saw_inner {


is it possible?

Yes, Expr::Unnest can arrive from the DataFrame API or a custom planner extension, not just from SQL. Keeping it as a defensive plan_err! so we never silently miscompile a mixed-mode plan

comphead · 2026-05-27T23:21:46Z

+##############################
+
+statement error DataFusion error: Error during planning: Cannot mix `unnest\(\.\.\.\)` \(or `explode\(\.\.\.\)`\) with `explode_outer\(\.\.\.\)` in the same SELECT
+SELECT unnest(xs), explode_outer(xs) FROM int_lists;


Spark actually doesnt support more than 1 generator per select

scala> spark.sql("select explode(array(1, 2, 3, null)), explode(array(1, 2, 3))").show(false) org.apache.spark.sql.AnalysisException: [UNSUPPORTED_GENERATOR.MULTI_GENERATOR] The generator is not supported: only one generator allowed per SELECT clause but found 2: "explode(array(1, 2, 3, NULL))", "explode(array(1, 2, 3))".

how about adding a datafusion.spark.allow_multiple_generators config to gate Spark's UNSUPPORTED_GENERATOR.MULTI_GENERATOR semantics?

comphead

@athlcode I'm planning to run the tests in Spark in see if there is any differences

…/explode_outer

comphead · 2026-06-02T17:45:52Z

    pub format: FormatOptions,
+    /// Spark-compatibility options (functions under `datafusion/spark` and
+    /// Spark-specific planner behavior)
+    pub spark: SparkOptions,


I think we should not be adding spark into the core, it would inverse the dependencies

comphead

Tests are fine, thanks @athlcode
Please help me to understand why we need to keep anything spark related in the core?

comphead · 2026-06-09T02:56:47Z

we might also update the SELECT doc like in #22672

…/explode_outer

athlcode · 2026-06-28T07:33:43Z

Tests are fine, thanks @athlcode Please help me to understand why we need to keep anything spark related in the core?

The mechanism has to live in core, but it isn't Spark-specific. NullHandling::PreserveAndExpandEmpty is just a new option on UnnestOptions in datafusion-common, where UnnestOptions already lives, and UnnestExec in datafusion-physical-plan is the only place the per-row logic can run. Expr::Unnest.outer is a new field on a core Expr variant, and downstream crates can't add fields to a pub enum variant in Rust, so the flag has to live where the variant is defined.

Neither references Spark by name. They're the generic mechanism for "unnest that preserves empty arrays as NULL rows," which Hive EXPLODE OUTER, Snowflake FLATTEN(OUTER => true), and BigQuery UNNEST ... WITH OFFSET all need too.

The Spark-flavored surface (the explode / explode_outer SQL aliases and the allow_multiple_generators config) is the only part that's truly dialect-specific. Happy to move those into datafusion-spark as a planner extension and a custom config registered via Extensions if you'd prefer the core diff to be purely the mechanism.

…/explode_outer

added NullHandling Enum matching Spark explode_outer

8460a80

github-actions Bot added sql SQL Planner core Core DataFusion crate common Related to common crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels May 10, 2026

github-actions Bot added the auto detected api change Auto detected API change label May 10, 2026

athlcode and others added 4 commits May 11, 2026 00:04

proto fix

5c16017

fixed rustdoc references

6783655

Merge branch 'main' of https://github.com/apache/datafusion into feat…

ca162c6

…/explode_outer

Merge branch 'main' into feat/explode_outer

6142aa0

comphead reviewed May 12, 2026

View reviewed changes

athlcode added 3 commits May 13, 2026 23:04

added test cases

745e641

Merge branch 'main' of https://github.com/apache/datafusion into feat…

17139b6

…/explode_outer

Merge branch 'feat/explode_outer' of https://github.com/KrishnaSudars…

eb3a489

…han7/datafusion into feat/explode_outer

Merge branch 'main' of https://github.com/apache/datafusion into feat…

e50c018

…/explode_outer

athlcode added 2 commits May 18, 2026 23:27

added explode_outer slt tests

71947a1

Merge branch 'main' of https://github.com/apache/datafusion into feat…

f133b1a

…/explode_outer

github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) labels May 18, 2026

athlcode added 2 commits May 27, 2026 18:47

Merge branch 'main' of https://github.com/apache/datafusion into feat…

3c2ac0f

…/explode_outer

format fix

6ba8e27

comphead reviewed May 27, 2026

View reviewed changes

comphead reviewed May 29, 2026

View reviewed changes

athlcode added 2 commits May 30, 2026 15:58

Merge branch 'main' of https://github.com/apache/datafusion into feat…

c6c6a6d

…/explode_outer

adding datafusion.spark.allow_multiple_generators config

7e789a2

github-actions Bot added the documentation Improvements or additions to documentation label May 30, 2026

comphead reviewed Jun 2, 2026

View reviewed changes

comphead changed the title ~~feat: Add support for Spark-compatible explode_outer function~~ feat: Add support for Spark-compatible explode_outer function for arrays Jun 3, 2026

Merge branch 'main' of https://github.com/apache/datafusion into feat…

f39b379

…/explode_outer

github-actions Bot removed the documentation Improvements or additions to documentation label Jun 28, 2026

updated SELECT doc

d1dc53f

github-actions Bot added the documentation Improvements or additions to documentation label Jun 28, 2026

athlcode added 5 commits June 28, 2026 13:10

fixed format

8c6909f

removed allow_multiple_generators

da4160f

fixed cargo fmt

1c0e522

Merge branch 'main' of https://github.com/apache/datafusion into feat…

55d0cb7

…/explode_outer

fixed slt tests

999399a

Uh oh!

Conversation

athlcode commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented May 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

athlcode commented May 13, 2026

Uh oh!

comphead commented May 14, 2026

Uh oh!

athlcode commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented May 15, 2026

Uh oh!

athlcode commented May 18, 2026

Uh oh!

kazuyukitanimura commented May 26, 2026

Uh oh!

athlcode commented May 27, 2026

Uh oh!

comphead May 27, 2026

Choose a reason for hiding this comment

Uh oh!

athlcode May 30, 2026

Choose a reason for hiding this comment

Uh oh!

comphead May 27, 2026

Choose a reason for hiding this comment

Uh oh!

athlcode May 30, 2026

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

comphead Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

comphead commented Jun 9, 2026

Uh oh!

athlcode commented Jun 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

athlcode commented May 10, 2026 •

edited

Loading

github-actions Bot commented May 10, 2026 •

edited

Loading

athlcode commented May 14, 2026 •

edited

Loading