OSDev.org

The Place to Start for Operating System Developers
It is currently Sat Nov 28, 2020 4:42 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: concurent build fails in VM guest env but succeeds in host
PostPosted: Thu Oct 22, 2020 1:22 pm 
Offline
Member
Member

Joined: Wed Nov 18, 2015 3:04 pm
Posts: 394
Location: San Jose San Francisco Bay Area
I am using combination of cmake and make utilities to compile large number of code comparable to the size of linux kernel. However it is becoming regular occurrence, when concurrent build is specified i.e. make -j`nproc` it belches out with lot of errors messages if I do in VM guest environment (in this case linux libvirtd/kvm) but same build completes OK in same system that is hosting the VM guest environment but on host O/S. I am not very knowledgeable about compiler architecture specially concurrent builds only have very basic understanding compiler back and front end. Based on the what I am observing any expert share their opinin on this?

_________________
key takeaway after spending yrs on sw industry: big issue small because everyone jumps on it and fixes it. small issue is big since everyone ignores and it causes catastrophy later. #devilisinthedetails


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Thu Oct 22, 2020 4:50 pm 
Offline
Member
Member

Joined: Tue Mar 04, 2014 5:27 am
Posts: 1058
If you don't build concurrently, do you still run into the same problem?


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Thu Oct 22, 2020 6:08 pm 
Offline
Member
Member

Joined: Wed Nov 18, 2015 3:04 pm
Posts: 394
Location: San Jose San Francisco Bay Area
alexfru wrote:
If you don't build concurrently, do you still run into the same problem?

that is very iffy and yes and no, meaning it is not consistent.
It is actually llvm toolset from llvm-project. I did notice for non-concurrent build (single thread), llvm project build OK. But there is variant of llvm-project which is part of ROCm AMD which, in single-thread in guest env, also failed and error was same.

I can not sure for certainty on my observation above, as once build working, I sort of moved on.
However guest environment build is bothering enough and definitely worth knowing.

Here is last few lines of build log, notice it says c++ internal compiler error:

[ 50%] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/TypeReferenceTracker.cpp.o
[ 50%] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/YAMLOutputStyle.cpp.o
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
lib/Target/AMDGPU/AsmParser/CMakeFiles/LLVMAMDGPUAsmParser.dir/build.make:62: recipe for target 'lib/Target/AMDGPU/AsmParser/CMakeFiles/LLVMAMDGPUAsmParser.dir/AMDGPUAsmParser.cpp.o' failed
make[2]: *** [lib/Target/AMDGPU/AsmParser/CMakeFiles/LLVMAMDGPUAsmParser.dir/AMDGPUAsmParser.cpp.o] Error 4
CMakeFiles/Makefile2:10265: recipe for target 'lib/Target/AMDGPU/AsmParser/CMakeFiles/LLVMAMDGPUAsmParser.dir/all' failed
make[1]: *** [lib/Target/AMDGPU/AsmParser/CMakeFiles/LLVMAMDGPUAsmParser.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 50%] Linking CXX static library ../../libLLVMSymbolize.a
[ 50%] Built target llvm-dwarfdump
[ 50%] Built target LLVMSymbolize
[ 50%] Linking CXX executable ../../bin/llvm-pdbutil
[ 50%] Built target llvm-pdbutil
[ 50%] Linking CXX static library ../libLLVMAnalysis.a
[ 50%] Built target LLVMAnalysis
[ 50%] Linking CXX executable ../../bin/obj2yaml
[ 50%] Built target obj2yaml
[ 50%] Linking CXX executable ../../bin/llvm-readobj
[ 50%] Built target llvm-readobj
Makefile:151: recipe for target 'all' failed
make: *** [all] Error 2

_________________
key takeaway after spending yrs on sw industry: big issue small because everyone jumps on it and fixes it. small issue is big since everyone ignores and it causes catastrophy later. #devilisinthedetails


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Thu Oct 22, 2020 6:19 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 2492
ggodw000 wrote:
c++: internal compiler error: Killed (program cc1plus)

Something is killing the compiler. Is it running out of memory?


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Fri Oct 23, 2020 4:11 am 
Offline
Member
Member

Joined: Wed Nov 18, 2015 3:04 pm
Posts: 394
Location: San Jose San Francisco Bay Area
not sure, may be that is possibility. the compile was no at least 46GB memory server on host. There were 8 same VM meaning at most 8GB mem allocated to each host.
Does compiling takes that much memory?

_________________
key takeaway after spending yrs on sw industry: big issue small because everyone jumps on it and fixes it. small issue is big since everyone ignores and it causes catastrophy later. #devilisinthedetails


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Fri Oct 23, 2020 12:36 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 711
ggodw000 wrote:
not sure, may be that is possibility. the compile was no at least 46GB memory server on host. There were 8 same VM meaning at most 8GB mem allocated to each host.
Does compiling takes that much memory?

Depends on language and features used. And on the source files, of course. Regrettably, C++ does have the features to make compiling arbitrarily difficult. Templates and constexpr are the two big ones driving memory cost.

_________________
Thou hast outraged, not insulted me, sir; but for that I ask thee not to beware of Starbuck; thou wouldst but laugh; but let Ahab beware of Ahab; beware of thyself, old man.


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Fri Oct 23, 2020 3:50 pm 
Offline
Member
Member

Joined: Tue Mar 04, 2014 5:27 am
Posts: 1058
ggodw000 wrote:
not sure, may be that is possibility. the compile was no at least 46GB memory server on host. There were 8 same VM meaning at most 8GB mem allocated to each host.
Does compiling takes that much memory?

I've once tried building TensorFlow on an ARM board with 3GB of RAM. I had to configure a swap file of like 8 or 16GB to build the monster. And even then I ran out of memory and built the last chunk and linked everything non-concurrently.
So, yeah, some things may be pretty demanding.


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Fri Oct 23, 2020 6:29 pm 
Offline
Member
Member

Joined: Wed Nov 18, 2015 3:04 pm
Posts: 394
Location: San Jose San Francisco Bay Area
i am going to try increase the VM memory and see if it works then.

_________________
key takeaway after spending yrs on sw industry: big issue small because everyone jumps on it and fixes it. small issue is big since everyone ignores and it causes catastrophy later. #devilisinthedetails


Top
 Profile  
 
 Post subject: Re: concurent build fails in VM guest env but succeeds in ho
PostPosted: Mon Oct 26, 2020 10:02 pm 
Offline
Member
Member

Joined: Wed Nov 18, 2015 3:04 pm
Posts: 394
Location: San Jose San Francisco Bay Area
it worked wonderfully. increased guest vm memory to 32-64G, and cpu-s to 32 core.

_________________
key takeaway after spending yrs on sw industry: big issue small because everyone jumps on it and fixes it. small issue is big since everyone ignores and it causes catastrophy later. #devilisinthedetails


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group