Help building the library

[nikhilkalige]

Hi..

I have been trying to build clanglite (python bindings) for clang 8 release, without using the conda infrastructure on a ubuntu 16.04 machine. I have made very little progress, so I would really appreciate any pointers that you could pass my way, to help me figure out the details.

Thank you..

[pfernique]

At which point are you blocked ?
Note that, in order to have working Python wrappers, you need to compile Clang and LLVM with the RTTI option (required for both Boost.Python and PyBind11 wrappers).
Moreover, since these wrappers are generated for the version 4.0.1 there is little chance that they will compile successfully against the 8.x.x versions without modifications.
I think the best is to regenerate wrappers using AutoWIG.
To this end, I used the bootstrap.py script to generated wrappers for Clang 4.0.1 with AutoWIG and Clang 3.8.1.
If you need more help, there are no problems, but it could be helpful to add pointer to the commit containing your current work and to add the script that you used to build the wrappers and its output.

[nikhilkalige]

I did build llvm and clang, but without the RTTI, I will rebuild it with that.

I updated the includes in cpp/tool.h file with new headers files from clang 8 release. Since then, I am trying to run bootstrap.py to regenerate the files, which I believe will update the src/py folder. I am getting this exception Path("'::boost::python' parameter"), may be because I did not install boost python, I thought I only needed pybind which I did install,

Relevant stack trace

['/clang/clanglite/src/cpp/tool.h']

/root/anaconda3/lib/python3.7/site-packages/clanglite/autowig_parser.py:896: Warning: 
  warnings.warn('', Warning)

Traceback (most recent call last):
  File "bootstrap.py", line 297, in <module>
    asg = autowig.controller(asg)
  File "bootstrap.py", line 62, in clanglite_controller
    for node in asg['::boost::python'].classes(nested = True):
  File "/root/anaconda3/lib/python3.7/site-packages/autowig-1.0.0-py3.7.egg/autowig/asg.py", line 1817, in __getitem__
    raise KeyError('\'' + node + '\' parameter')
KeyError: Path("'::boost::python' parameter")

[nikhilkalige]

I disabled boost and I was able to make some progress. The script wrote to _clanglite.cpp and _clanglite.h, very early in the process (may be like 10 mins into) and then continued to run for a few hours without writing to the file system, and I killed it after that.

[pfernique]

I switch from Boost.Python to PyBind11 not so long ago so I did missed some things in the bootstrap.py file. Disabling Boost.Python was the right thing to do.

When wrapping Clang, the AutoWIG process is still long (1 hour) since:

• I didn't parallelized file writing.
• Recursive requests are made with no cache to see if objects are meant to be wrapped or not.

I will work on those two things probably next week but in the meantime, the process will be slow.
But more than one hour for all files seems to long (file writing is taking at most 2 minutes for classes with big amount of methods such as class ::clang::Decl).
Probably the best thing to do is to launch an ipython console, run the script (%run bootstrap.py) and kill it from within after some time to see at which point it is blocked.

[nikhilkalige]

As I said it took a long time, I just let it run for 96h 28min, it just finished now, let me see what output it has generated.

[pfernique]

Wahou... With 96h 28min, it is really faster than manually ? Well at least you can do other things...
Still, did you published it on GitHub ? This process will be really good to test against later optimizations of AutoWIG.
Nevertheless, it is really strange that it is taking so much time. If I can reproduce it, I will take a look at this process using a profiler.

[nikhilkalige]

Few locations where it took a long time are in decoraters.write, the section with for export in modules.exports, typdefs took like 2 days. However, I tried compiling now, but I am having trouble compiling it, as gcc is throwing a errors with the very first file. It has the following contents

#include "_clanglite.h"

bool  (*function_pointer_00011be0053e5e36b3dbda4d7cdf9181)(class ::clang::Attr * const &) = ::llvm::isa;

void wrapper_00011be0053e5e36b3dbda4d7cdf9181(pybind11::module& module)
{
    module.def("isa", function_pointer_00011be0053e5e36b3dbda4d7cdf9181, "");
}

[nikhilkalige]

Some random changes I made to able to run it are here nikhilkalige@dcb6d73

[pfernique]

Thanks for the link.
Could you post the gcc error ?

[nikhilkalige]

Log for scons py build

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o build/py/wrapper_00011be0053e5e36b3dbda4d7cdf9181.os -c -ffunction-sections -fdata-sections -Wno-deprecated-declarations -I /home/lonewolf/softwares/clanglite/pybind11/include/ -I /usr/include/python3.5m/ -I /home/lonewolf/softwares/clanglite/llvm-project/build/tools/clang/include/ -I /home/lonewolf/softwares/clanglite/llvm-project/clang/include/ -I /home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/ -I /home/lonewolf/softwares/clanglite/llvm-headers/llvm-c-8/ -I ../../ -fvisibility=hidden -Wl,--no-undefined -fdiagnostics-color=always -Winvalid-pch -fPIC -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DPYBIND11_GENERATOR -I/home/lonewolf/.virtualenvs/clanglite/include -I/home/lonewolf/.virtualenvs/clanglite/include/python3.6m build/py/wrapper_00011be0053e5e36b3dbda4d7cdf9181.cpp
build/py/wrapper_00011be0053e5e36b3dbda4d7cdf9181.cpp:3:101: error: no matches converting function 'isa' to type 'bool (*)(class clang::Attr* const&)'
 bool  (*function_pointer_00011be0053e5e36b3dbda4d7cdf9181)(class ::clang::Attr * const &) = ::llvm::isa;
                                                                                                     ^~~
In file included from /home/lonewolf/softwares/clanglite/llvm-project/clang/include/clang/Basic/LLVM.h:22,
                 from /home/lonewolf/softwares/clanglite/llvm-project/clang/include/clang/Basic/LangOptions.h:19,
                 from /home/lonewolf/softwares/clanglite/llvm-project/clang/include/clang/Format/Format.h:18,
                 from ../../clang/clanglite/src/cpp/tool.h:13,
                 from build/py/_clanglite.h:6:
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h:142:56: note: candidate is: 'template<class X, class Y> bool llvm::isa(const Y&)'
 template <class X, class Y> LLVM_NODISCARD inline bool isa(const Y &Val) {

[pfernique]

That's probably a bug related to explicit specializations of function templates: AutoWIG sees that this specialization exists and treat it as a normal function whereas it should be treated as a template function.
Does it pass if you modify the line

bool  (*function_pointer_00011be0053e5e36b3dbda4d7cdf9181)(class ::clang::Attr * const &) = ::llvm::isa;

to

bool  (*function_pointer_00011be0053e5e36b3dbda4d7cdf9181)(class ::clang::Attr * const &) = ::llvm::isa<::clang::Attr *, ::clang::Attr *>;

[nikhilkalige]

/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h: In instantiation of ‘static bool llvm::isa_impl<To, From, Enabler>::doit(const From&) [with To = clang::Attr*; From = clang::Attr; Enabler = void]’:
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h:107:36:   required from ‘static bool llvm::isa_impl_cl<To, const From*>::doit(const From*) [with To = clang::Attr*; From = clang::Attr]’
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h:133:40:   required from ‘static bool llvm::isa_impl_wrap<To, FromTy, FromTy>::doit(const FromTy&) [with To = clang::Attr*; FromTy = const clang::Attr*]’
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h:124:60:   required from ‘static bool llvm::isa_impl_wrap<To, From, SimpleFrom>::doit(const From&) [with To = clang::Attr*; From = clang::Attr* const; SimpleFrom = const clang::Attr*]’
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h:144:74:   required from ‘bool llvm::isa(const Y&) [with X = clang::Attr*; Y = clang::Attr*]’
build/py/wrapper_00011be0053e5e36b3dbda4d7cdf9181.cpp:5:101:   required from here
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/Support/Casting.h:59:23: error: ‘classof’ is not a member of ‘clang::Attr*’

I looked at the source, the classof is defined only for the object that inherit from Attr, ex: TypeAttr

[pfernique]

Seems to require either ::llvm::isa<::clang::Attr, ::clang::Attr *> or ::llvm::isa<::clang::Attr *, ::clang::Attr>.
But perhaps, the better is to comment it

#include "_clanglite.h"

//bool  (*function_pointer_00011be0053e5e36b3dbda4d7cdf9181)(class ::clang::Attr * const &) = ::llvm::isa;

void wrapper_00011be0053e5e36b3dbda4d7cdf9181(pybind11::module& module)
{
    //module.def("isa", function_pointer_00011be0053e5e36b3dbda4d7cdf9181, "");
}

It didn't have much time to look at it this week, perhaps the next one but it's definitely a bug of ClangLite parser and this function should not be wrapped.

[nikhilkalige]

Thanks for that fix, ::llvm::isa<::clang::Attr, ::clang::Attr *> does the job, but it looks like it throws a lot of more compiler errors.

This i fixed

build/py/wrapper_00150170c77b5101840858c2720f291b.cpp:4:127: error: no matches converting function ‘cast’ to type ‘const class clang::ReturnStmt* (*)(const class clang::Stmt*)’
 class ::clang::ReturnStmt const * (*function_pointer_00150170c77b5101840858c2720f291b)(class ::clang::Stmt const *) = ::llvm::cast<class ::clang::ReturnStmt const*, class ::clang::Stmt const *>;

But a few more showed up, after that

build/py/wrapper_001f5ad35d9d5097868915c17cd0971e.cpp:5:174: error: no matches converting function 'getValue' to type 'const enum class clang::LangAS& (class llvm::Optional<clang::LangAS>::*)() const'
 st & (::llvm::Optional< enum ::clang::LangAS >::*method_pointer_a7c5f57c3c7b57c7b843da289f25f407)()const= &::llvm::Optional< enum ::clang::LangAS >::getValue;
                                                                                                                                                      ^~~~~~~~

In file included from /home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/ADT/STLExtras.h:20,
                 from /home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/ADT/StringRef.h:13,
                 from /home/lonewolf/softwares/clanglite/llvm-project/clang/include/clang/Basic/ObjCRuntime.h:19,
                 from /home/lonewolf/softwares/clanglite/llvm-project/clang/include/clang/Basic/LangOptions.h:20,
                 from /home/lonewolf/softwares/clanglite/llvm-project/clang/include/clang/Format/Format.h:18,
                 from ../../clang/clanglite/src/cpp/tool.h:13,
                 from build/py/_clanglite.h:6:
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/ADT/Optional.h:177:7: note: candidates are: 'T&& llvm::Optional<T>::getValue() && [with T = clang::LangAS]'
   T &&getValue() && { return std::move(*getPointer()); }
       ^~~~~~~~
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/ADT/Optional.h:162:6: note:                 'T& llvm::Optional<T>::getValue() & [with T = clang::LangAS]'
   T &getValue() LLVM_LVALUE_FUNCTION { return *getPointer(); }
      ^~~~~~~~
/home/lonewolf/softwares/clanglite/llvm-headers/llvm-8/llvm/ADT/Optional.h:161:12: note:                 'const T& llvm::Optional<T>::getValue() const & [with T = clang::LangAS]'
   const T &getValue() const LLVM_LVALUE_FUNCTION { return *getPointer(); }
            ^~~~~~~~
build/py/wrapper_001f5ad35d9d5097868915c17cd0971e.cpp:6:163: error: no matches converting function 'getValue' to type 'enum class clang::LangAS& (class llvm::Optional<clang::LangAS>::*)()'
 :LangAS & (::llvm::Optional< enum ::clang::LangAS >::*method_pointer_b4a83a8ee3ad5e6493e15cd15bf15553)()= &::llvm::Optional< enum ::clang::LangAS >::getValue;
                                                                                                                                                      ^~~~~~~~

So will try again this weekend may be..

[pfernique]

The best thing is perhaps to use the code I have done for the wrapping of Eigen.

subprocess.call(['scons', '-j' + jobs, '-k', '--diagnostics-color=never'])

s = subprocess.Popen(['scons', 'autowig', '-j' + jobs, '-k', '--diagnostics-color=never'], stderr=subprocess.PIPE)
out, err = s.communicate()

autowig.feedback.plugin = 'comment'
prev = None
curr = err
while not prev == curr:
    if not prev == curr:
        prev = curr
        autowig.feedback(curr, '.', asg, variant_dir=variant_dir,
                                         src_dir=src_dir)
        s = subprocess.Popen(['scons', '-j1', '-k', '--diagnostics-color=never'], stderr=subprocess.PIPE)
        out, curr = s.communicate()
    else:
        print("done")

This will use the feedback of autowig that is commenting lines causing errors in wrappers.
To used it you need to:

• Load your asg using pickle:

    with open(os.path.join(autowig.__path__[0], '..', 'scons_tools', 'site_autowig', 'ASG', 'clanglite.pkl'), 'rb') as filehandler:
    asg = pickle.load(filehandler)

• Define the number of jobs (usual number given to scons).

• Define directories where sources and builds can be found:

  variant_dir = 'build/src'
  src_dir = 'src'