fix: Native engine crashes on literal DateTrunc and TimestampTrunc#3668
fix: Native engine crashes on literal DateTrunc and TimestampTrunc#36680lai0 wants to merge 3 commits intoapache:mainfrom
Conversation
|
Broadcasting scalars to arrays to fit existing API limitations seems like a hack. It also means in the (scalar, scalar) case we return an array, when we could also return a scalar. |
|
Thanks @mbutrovich for the reminder—I’ve updated the PR. |
andygrove
left a comment
There was a problem hiding this comment.
There are already SQL file tests for this exact scenario in trunc_date.sql and trunc_timestamp.sql. The literal queries on lines 36-37 and 39-40 respectively are currently marked with ignore(https://github.com/apache/datafusion-comet/issues/3342), which is the issue this PR closes. It would be great to remove those ignore annotations so the existing SQL tests cover this fix. That would also mean the new Scala tests in CometTemporalExpressionSuite might not be needed since the SQL file tests already cover multiple format strings (year, month, quarter, day) rather than just year.
|
Thanks @andygrove comment. PR updated. |
Which issue does this PR close?
Closes #3342
Rationale for this change
When ConstantFolding is disabled, literal trunc() and date_trunc() calls reach the native engine with both arguments as Scalar. The existing match arms only handled (Array, Scalar) and (Array, Array) , so the (Scalar, Scalar) case fell through to the error branch and crashed.
What changes are included in this PR?
Use into_array(num_rows) to convert the date argument to an Array before matching on format, so all input combinations are handled uniformly.
Convert the timestamp argument to an Array via into_array(num_rows) before matching on format.
How are these changes tested?
Added test cases in
CometTemporalExpressionSuitethat disable ConstantFolding and verify literaltrunc()anddate_trunc()queries execute correctly on Comet and produce results matching Spark.