Skip to content

feat(isthmus): configurable fallback to dynamic function mapping#647

Open
ZorinAnton wants to merge 6 commits intosubstrait-io:mainfrom
ZorinAnton:fallback-to-dynamic-function-mappings
Open

feat(isthmus): configurable fallback to dynamic function mapping#647
ZorinAnton wants to merge 6 commits intosubstrait-io:mainfrom
ZorinAnton:fallback-to-dynamic-function-mappings

Conversation

@ZorinAnton
Copy link
Contributor

This PR introduces a new autoFallbackToDynamicFunctionMapping feature flag that enables automatic support for all unmapped extension functions (scalar, aggregate, and window functions) in SQL↔Substrait conversions.

Copy link
Member

@vbarua vbarua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working on a PR (#649) to try and make wiring in what you're doing here easier. We're starting to run into the limits of the existing design of the function conversion system, and I'm wary of adding more complexity before cleaning it up.

My big issue is with how we're using the feature flags for this, because we're embedding a ton of conditional configuration logic into constructors all over the codebase, which makes it very difficult to reason about an already difficult part of the system. What I would like to do is separate out the configuration from the constructors. Effectively, can we construct all the function converters we need outside of the SubstraitRelNode and SubstraitRelVisitor, and then inject them in.

@ZorinAnton ZorinAnton force-pushed the fallback-to-dynamic-function-mappings branch 3 times, most recently from 9314af1 to 956b447 Compare March 6, 2026 06:59
@ZorinAnton ZorinAnton marked this pull request as ready for review March 9, 2026 11:09
@ZorinAnton ZorinAnton requested a review from vbarua March 9, 2026 11:10
@nielspardon
Copy link
Member

@ZorinAnton can you rebase one more time and resolve the conflicts? thanks

@ZorinAnton ZorinAnton force-pushed the fallback-to-dynamic-function-mappings branch from a5c465f to 66ed907 Compare March 13, 2026 13:27
@vbarua
Copy link
Member

vbarua commented Mar 13, 2026

High-level question

What's the value of having a separate AutomaticDynamicFunctionMappingConverterProvider instead of expanding the DynamicConverterProvider. Put another way, would it make sense to just have a non-dynamic mode (ConverterProvider) and dynamic mode (DynamicConverterProvider)?

@ZorinAnton
Copy link
Contributor Author

While they use similar mechanics under the hood, they serve two distinct use cases:

  • DynamicConverterProvider is specifically for custom UDFs injected at runtime. It explicitly filters out the standard Substrait catalog.
  • AutomaticDynamicFunctionMappingConverterProvider is a fallback for standard Substrait functions (like strftime) that exist in the official YAMLs but haven't been manually added to our static FunctionMappings.java yet.

Keeping them separate allows users to opt into custom UDF support, the standard function fallback, or both independently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants