refac: Use consistent "outer" default alignment for all arithmetic operations#590
refac: Use consistent "outer" default alignment for all arithmetic operations#590FBumann wants to merge 3 commits intoharmonize-linopy-operationsfrom
Conversation
Replace the shape-dependent heuristic (same-shape → "override", different-shape → "left"/"outer") with a uniform "outer" default for all arithmetic (+, -, *, /). This fixes two bugs: 1. Same-shape operands with different coords were silently matched by position 2. Addition was not associative: (y + factor) + x != y + (x + factor) Key changes: - merge(): Remove check_common_keys_values heuristic, always use "outer" - _align_constant(): Default to "outer", fill expression const with 0 and operand with operation-dependent fill_value separately - _apply_constant_op(): Fill NaN coefficients with 0 before applying operation to prevent NaN propagation from reindex fill - to_constraint(): Handle DataArray RHS directly (without sub) to preserve intentional NaN masking; default remains "left" for constraints - Variable.__mul__(): Use _multiply_by_constant() instead of to_linexpr(coeff) so a*factor and (1*a)*factor produce the same result
Change multiplication fill_value from 0 to 1 to match division, which already uses 1. This makes fill values consistent with algebraic identity elements: 0 for addition/subtraction, 1 for multiplication/division. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use "inner" join as the default for all arithmetic operations, matching xarray's own arithmetic_join default. This eliminates the need for fill values entirely — only shared coordinates appear in results. Disjoint coordinates produce empty results, making mistakes immediately visible. Users can opt in to union behavior with explicit join="outer". Constraint DataArray RHS default remains "left". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
thanks @FBumann, just FYI, the shape dependent overwrite was introduced as a convention and the documentation should state is clearly, but I also understand that this is not very intuitive and user might be confused. Since changing the behavior is very breaking (in pypsa for example we rely in the overwrite mechanism) I would like to make a clear transition path to the new behavior. So, we could pull in #572 and then add a global options tag in linopy to allow for new arithmetic alignments which users to opt in. default then in next major version, likely v2 ? |
First of all, this PR is only a sketch up. |
Breaking Change
This PR changes the default coordinate alignment for all arithmetic operations from a shape-dependent heuristic to a consistent
"inner"join (intersection of coordinates). This matches xarray's ownarithmetic_joindefault. Users who relied on the old positional matching for same-shape operands with different coordinates will see reduced or empty results instead. The old behavior is available via explicitjoin="override".Convention
All arithmetic (
+,-,*,/) defaults to"inner"(intersection of coordinates).This is the only choice that:
(a + b) + c == a + (b + c)regardless of operand shapesarithmetic_joindefault is"inner"No fill values are needed with inner join — every position in the result has data from both operands.
Constraints with DataArray RHS default to
"left"— the expression defines where variables exist; missing RHS values becomeNaN(masked out).+,-,*,/)"inner""left""inner"sub, inherits the arithmetic defaultWhat changes from the status quo
The primary behavioral change is when operands have the same shape but different coordinates:
x(i=[0,1,2]) + z(i=[5,6,7])"override": 3 entries, positional"inner": 0 entries, empty (visible error)x(i=[0,1,2]) + c(i=[5,6,7])"override": 3 entries, positional"inner": 0 entries, empty (visible error)x(i=[0,1,2]) * c(i=[5,6,7])"override": 3 entries, positional"inner": 0 entries, empty (visible error)When operands have subset/superset relationships (e.g., x has 20 coords, subset has 2):
When operands have matching coordinates: no change.
Available join modes
For explicit control, use
.add(),.sub(),.mul(),.div(),.le(),.ge(),.eq()with ajoinparameter:join"inner"(default)"outer""left""right""override""exact"Why this is necessary
The previous heuristic (
check_common_keys_values) chose alignment based on operand shapes:"override"(positional matching, ignoring labels)"outer"(for expr+expr) or"left"(for expr+constant)This caused two bugs:
x(i=[0,1,2]) + z(i=[5,6,7])matched by position even though coordinates were entirely different, producing wrong results underx's labels(y + factor) + x != y + (x + factor)because"left"for expr+constant dropped the constant's extra coordinates before they could be recovered by a subsequent expr additionChanges
linopy/expressions.pymerge(): Removecheck_common_keys_valuesheuristic; whenjoinisNone, always use"inner". Pre-pad helper dimensions (_term) when concatenating along non-helper dimensions to ensure inner join only affects coordinate dimensions._align_constant(): Default to"inner"; usexr.alignfor non-override/non-left joins_apply_constant_op(): Fill NaN coefficients with 0 before applying op (prevents NaN propagation from reindex fill at new positions)_multiply_by_constant(): Use fill_value=1 (identity element) for explicit outer/left joins_divide_by_constant(): Use fill_value=1 (identity element) for explicit outer/left joinsto_constraint(): Handle DataArray RHS directly (withoutsub) to preserve intentional NaN masking for constraint positions; supportsjoinparameter for all join modeslinopy/variables.pyVariable.__mul__(): Useto_linexpr()._multiply_by_constant(other)instead ofto_linexpr(other)soa * factorand(1 * a) * factorproduce identical resultstest/test_linear_expression.pyTestSubsetCoordinateAlignmentfor inner defaults (subset operations give intersection)test_linear_expression_sumfor inner default on disjoint slices (empty result)TestAssociativitytests for inner semantics; addtest_outer_gives_unionTestJoinParameterquadratic tests for inner default on cross-product expressionstest/test_optimization.pytest_non_aligned_variablesfor inner default (uncovered variables have no constraint)examples/coordinate-alignment.ipynb"inner"for all arithmetic,"left"for constraint DataArray RHSjoin="outer"as explicit opt-in for union behaviorChecklist
doc/release_notes.rstof the upcoming release is included