Upgrade to Python 3 support

[jmw182]

Official Python 2.7 support is ending in January 2020. I have been working on upgrading our version of PyRTMA to use Python 3.7 with a cross-platform replacement for ctypeslib and ctypesgen that is Python 3 compatible.

I have had luck using ctypeslib2, which depends on the LLVM Clang compiler (which is the default compiler on Mac and can be installed on Windows/Linux) and the python Clang support package, both available through PyPi (aka pip). The h2xml and xml2py scripts previously used on Linux can be replaced by clang2py from ctypeslib2, which also works on Windows, replacing ctypesgen.

The biggest issue I have encountered is parsing multi-line macro definitions (e.g. DF_MSG_HEADER_FIELDS in Dragonfly_types.h) with ctypeslib2. I submitted an issue to ctypeslib2, but in the meantime a workaround is to remove the DF_MSG_HEADER_FIELDS definition and replace all references to it with the defined value.

[hongweimao]

@jmw182 Were you able to build message definitions for Python3 using this approach? I got errors due to some function-like macros used in C header files, like check_flag_bits() in common_defs.h.
I'm using Python 3.7.3 + ctypeslib2 2.2.2 + clang 6.0.0.2 + LLVM 8.0.0 in Windows 10 64-bit.

[jmw182]

I have only tested with the example message definitions so far, but yeah I suspect macro functions would not work. I don't think we have a file "common_defs.h" in our system. Removing the -k m input to clang2py might resolve the error but would only parse the MDF structures, not MIDs, MT numbers, or constants. It seems like the clang2py macro parser can only handle simple constant definitions (e.g. #define MID_MESSAGE_MANAGER 0).

[jmw182]

If ctypeslib2/clang2py isn't going to be a robust solution, then there may be alternatives. The "official" ctypesgen isn't maintained, but this fork claims to be python 3 compatible and is worth investigating: https://github.com/olsonse/ctypesgen

The Dragonfly installation instructions say that "ctypesgen" was used on Windows and "ctypeslib" was used on Linux, but I don't see anything that suggests ctypesgen isn't cross platform.

[jmw182]

I tried using clang2py with our full set of message definitions and was able to parse them and do a quick test with reading messages from Matlab.

I'm calling clang2py like this: python3 %CLANG2PY% -c -k cdefmstu -i -o climber_config3.py C:\git\climber\include\climber_config.h (our BCI system is now called climber)

I tried out the olsonse/ctypesgen fork above but didn't have much luck. I've done some research, and it doesn't seem like there are many options for parsing C macros in python, most of these tools rely on a preprocessor (like clang) to process the code before translating to python. So while it's somewhat limited compared to the older ctypesgen, this clang2py seems to be the simplest option for python3 that I've found.

[hongweimao]

Clang2py was able to parse my message definitions (with -k cdefmstu) after removing those macro functions. I'll find out if any module uses those macro functions in future testing.

I had no luck with the olsonse/ctypesgen fork either. I didn't figure out how to use it because ctypesgen.py was missing.

[jmw182]

Glad to hear. With the ctypesgen fork, I tried running the "run.py" function instead of "ctypesgen.py" but I still ran into errors.

I think my plan for now will be to attempt to maintain two parallel versions of PyRTMA and python message defs, one for Python 2.7 and one for Python 3.7 (calling them something like PyRTMA2 and PyRTMA3) since some of our other dependencies don't support Python 3 yet either (but I expect to encounter APIs/tools that won't support Python 2.7 soon). I imagine it won't be too hard to update the project/batch files to generate both versions at the same time.

[ediril]

It looks like ctypesgen has been updated to support python3. I originally used ctypesgen only on windows since ctypeslib was available on linux but not on windows. But ctypesgen seems to be cross-platform and now that ctypeslib is very old/unmaintained, we might as well use ctypesgen on all platforms going forward. I pushed some changes to dragonfly fork in my github account if you guys want to have a look.

[jmw182]

Thanks Emrah, I'll have to try out the updated ctypesgen when I get a chance. One thing I noticed when we switched to the clang-based ctypeslib2 was that the generated .py files looked a lot cleaner and more human-readable than the python 2 ctypesgen produced .py files. That said if there is no strict compiler dependency and it supports multi-line macro definitions (like DF_MSG_HEADER_FIELDS), then that would be an improvement over ctypeslib2.

[ediril]

Right, ctypeslib2 does not support multi-line macro definition as you mentioned, also it does not support function macros (such as #define set_flag_bits( A, bitmask) ((A) |= (bitmask))). It looks like they are features the author just did not implement (yet?). I tried to have a look at it over the weekend but it requires decent understanding of the clang parser. I spent some time chasing the error into the guts of clang python bindings but didn't have enough time to grok it enough and add the missing features.

ctypesgen does produce a much larger output. It's meant for wrapping C++ code (not just header files) to be able to use it with Python, so it's more of a SWIG replacement. We really don't need all the extra stuff it puts in the .py files since we need a subset of the features. But there doesn't seem to be a way of excluding them via a command line argument unfortunately. Still, the extra stuff shouldn't hurt.

As I mentioned already, I incorporated it into my fork of the dragonfly repo and tried the python examples. They worked for me both on windows and linux. I also used ctypesgen to translate Dragonfly_config.h in the BCI repo which worked but I didn't try running modules with the Dragonfly_config.py file it produced.

Over the weekend, I also tried building dragonfly on OSX. I did succeed but I got a seg-fault as soon as I ran MessageManager. I did not look into what the problem was. I was hoping it would just work. To get it to build, I had to adjust some of the platform specific #ifdefs. It would be nice to get it working on OSX but not very high on the todo list.

[jmw182]

I have our build of rtma working on Mac OSX. Our repo isn't a direct fork of this Dragonfly repo (it's a private repo named rtma and came from our old svn repo), but it's pretty similar. I will add you as a collaborator so you can see the changes I made (in commit 66f154862 on 2/21/19, I made changes related to socket options).

[ediril]

Thanks, I'll check it out