Skip to content

std::string overloads#5995

Open
bwehlin wants to merge 6 commits intopybind:masterfrom
bwehlin:std_string_overloads
Open

std::string overloads#5995
bwehlin wants to merge 6 commits intopybind:masterfrom
bwehlin:std_string_overloads

Conversation

@bwehlin
Copy link

@bwehlin bwehlin commented Feb 25, 2026

I've added some std::string overloads for various functions related to modules.

The motivation is the following. My project has a lot of code like

  template <typename T>
  void register_functions(py::module_& m, const std::string& suffix)
  {
    m.def(("func1_" + suffix).c_str(), &func1<T>);
    m.def(("func2_" + suffix).c_str(), &func2<T>);
  }

  void register_np_conversions(py::module_& m)
  {
    register_functions<float>(m, "32");
    register_functions<double>(m, "64");
  }

With the current pybind11, I have to write m.def(("func1_" + suffix).c_str(), ...), but it would be nicer to be able to do just m.def("func1_" + suffix, ...).

bwehlin and others added 6 commits February 25, 2026 11:35
```
g++ -o pybind11/tests/test_modules.os -c -std=c++20 -fPIC -fvisibility=hidden -O0 -g -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wundef -Wnon-virtual-dtor -Wunused-result -Werror -funsigned-char -Wpedantic -isystem /wrk/cpython_installs/v3.14.2_df793163d58_default/include/python3.14 -isystem /usr/include/eigen3 -DPYBIND11_SMART_HOLDER_PADDING_ON -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD_IF_AVAILABLE -DPYBIND11_TEST_BOOST -Ipybind11/include -I/wrk/forked/pybind11/include -I/wrk/clone/pybind11/include /wrk/forked/pybind11/tests/test_modules.cpp
In file included from /wrk/forked/pybind11/include/pybind11/eval.h:14,
                 from /wrk/forked/pybind11/tests/pybind11_tests.h:3,
                 from /wrk/forked/pybind11/tests/constructor_stats.h:68,
                 from /wrk/forked/pybind11/tests/test_modules.cpp:11:
/wrk/forked/pybind11/include/pybind11/pybind11.h: In instantiation of ‘pybind11::module_& pybind11::module_::def(const std::string&, Func&&, const Extra& ...) [with Func = test_submodule_modules(pybind11::module_&)::<lambda(int)>; Extra = {pybind11::arg}; std::string = std::__cxx11::basic_string<char>]’:
/wrk/forked/pybind11/tests/test_modules.cpp:129:21:   required from here
/wrk/forked/pybind11/include/pybind11/pybind11.h:1549:57: error: binding reference of type ‘std::remove_reference<pybind11::arg>::type&’ {aka ‘pybind11::arg&’} to ‘const pybind11::arg’ discards qualifiers
 1549 |         return def(name_.c_str(), f, std::forward<Extra>(extra)...);
      |                                      ~~~~~~~~~~~~~~~~~~~^~~~~~~
In file included from /usr/include/c++/13/bits/stl_pair.h:61,
                 from /usr/include/c++/13/bits/stl_algobase.h:64,
                 from /usr/include/c++/13/bits/specfun.h:43,
                 from /usr/include/c++/13/cmath:3699,
                 from /usr/include/c++/13/math.h:36,
                 from /wrk/cpython_installs/v3.14.2_df793163d58_default/include/python3.14/Python.h:23,
                 from /wrk/forked/pybind11/include/pybind11/conduit/wrap_include_python_h.h:44,
                 from /wrk/forked/pybind11/include/pybind11/detail/common.h:12,
                 from /wrk/forked/pybind11/include/pybind11/attr.h:13,
                 from /wrk/forked/pybind11/include/pybind11/detail/class.h:12,
                 from /wrk/forked/pybind11/include/pybind11/pybind11.h:12:
/usr/include/c++/13/bits/move.h:70:56: note:   initializing argument 1 of ‘constexpr _Tp&& std::forward(typename remove_reference<_Functor>::type&) [with _Tp = pybind11::arg; typename remove_reference<_Functor>::type = pybind11::arg]’
   70 |     forward(typename std::remove_reference<_Tp>::type& __t) noexcept
      |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~
```
@rwgk
Copy link
Collaborator

rwgk commented Mar 2, 2026

I started looking at this on the weekend but didn't get very far. I pushed a new tests (generated by Cursor) that I believe will make the builds here fail (but should work with master; although I haven't validated that).

I probably won't have time to come back here this week.

Wild idea on my mind: maybe we should drop C++11 and C++14 first (that's overdue now IMO), then use string_view? But I'm not even sure if that idea has merit; just putting it out here in case someone else has more time to look into it.

@rwgk
Copy link
Collaborator

rwgk commented Mar 2, 2026

Leaving this here, JIC there is something useful in it:


Your first prompt

Could you please review this PR (current branch): /wrk/logs/gh_pr_info_pybind_pybind11_5995_2026-02-28+214859.md

Did the PR author add the overloads systematically? Do you see any obvious omissions?

Is the test coverage complete?

Cursor GPT-5.3 Codex (Extra High) response

Findings (ordered by severity)

  • High: module_::def(std::string, ...) is currently broken for common Extra args like py::arg.
    The new overload forwards extra with std::forward<Extra>(extra) even though parameters are const Extra&, which causes a constness mismatch and compile failure.
template <typename Func, typename... Extra>
module_ &def(const std::string &name_, Func &&f, const Extra &...extra) {
    return def(name_.c_str(), f, std::forward<Extra>(extra)...);
}

I confirmed with a compile-only snippet using m.def(std::string("f"), ..., py::arg("x")); compiler error was:
binding reference of type 'pybind11::arg&' to 'const pybind11::arg' discards qualifiers.

  • Medium: same overload drops perfect-forwarding for Func (f passed as lvalue), regressing move-only callables.
    m.def("f", [p = std::move(unique_ptr)]{...}) compiles today via the const char* overload, but the std::string overload fails because it passes f as an lvalue into the downstream template (copy required, copy ctor deleted).

  • Low: overload additions are not fully systematic across the module_ surface.
    The deprecated module_(const char*, const char*) constructor has no std::string counterpart, and create_extension_module only got a (std::string, std::string, ...) form (no mixed/null-doc variants). Might be intentional, but it is asymmetric with existing overload patterns.

Obvious omissions

  • No tests for def(std::string, ..., py::arg(...)) (this would have caught the high-severity bug).
  • No tests for move-only callable with def(std::string, ...) (would catch the forwarding regression).
  • No direct tests for add_object(std::string, ...).
  • No direct tests for create_extension_module(std::string, std::string, ...).
  • No test for def_submodule(std::string, std::string) (docstring path).
  • import(std::string) is only exercised on failure path, not success path.

Test coverage assessment

  • Not complete. The added tests only cover:
    • def_submodule(std::string) and def(std::string) in a basic no-Extra case.
    • import(std::string) in an exception path.
  • They do not cover the new overloads comprehensively or the template edge cases that matter most.

@bwehlin
Copy link
Author

bwehlin commented Mar 3, 2026

I started looking at this on the weekend but didn't get very far. I pushed a new tests (generated by Cursor) that I believe will make the builds here fail (but should work with master; although I haven't validated that).

I probably won't have time to come back here this week.

Wild idea on my mind: maybe we should drop C++11 and C++14 first (that's overdue now IMO), then use string_view? But I'm not even sure if that idea has merit; just putting it out here in case someone else has more time to look into it.

Thank you @rwgk for looking at this issue adding the tests. I'm also a bit busy this week but I will try to get to fixing the issue(s) ASAP.

(I also thought about string_view but I didn't want to take a stance on which C++ standard pybind11 should require ;) -- perhaps one could #ifdef __cpp_lib_string_view?)

@rwgk
Copy link
Collaborator

rwgk commented Mar 3, 2026

The string_view idea is motivated by the hope that we don't have to add overloads, i.e. this cost-benefit opinion:

Engineering away the need to write (...).c_str() is a really marginal, cosmetic improvement. Paying for that by adding a whole family of overloads that needs to be maintained is a high cost in comparison.

My thinking from there:

  • I really think it's time to drop C++ 11 and 14. We're already marking master as v3.1 (minor version bump from last release), so we could do that now.
  • With that done, maybe we can simply replace the existing const char * types with std::string_view and we're done?

@bwehlin
Copy link
Author

bwehlin commented Mar 3, 2026

If pybind11 is moving to C++17, having just string_view is clearly the better option 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants