Thread migration. How?

[AlexHully]

Hi,

I currently have an issue with thread migration.

My system has numa and 4 cores.
My goal: for a very long calculation, I wish to divide it into multiple physical cores (let's say 2 to keep things "simple").

Now, I am interested to understand it at the kernel level.

From what I understood, I will have to respect those points:
1) Locate the thread.
2) Send an IPI to the core I wish.
3) Copy from core1 to core2 (?)
4) Send results back (should we create a specific interrupt?)

Now, Numa makes me scratch my head. Core 1 has the process running. It does so in its numa frame. Core 2 has another numa frame.

1) What are the practical modifications I will have to make to my process image to fit the numa constraint on this new core?
2) How is handled the copy? Will I have to call memcpy to copy from core1 to core2? (that could be slow even with optimized code)
3) Will I have to deal with Thread Local Storage at any point?
4) Once the job is done, how to retrieve the results? IPC? IPI back? (Isn't it slow?)

The overall administration seems very heavy and possibly relatively slow (difficult to scale linearly, but rather with diminishing return).

Thread migration is a difficult problem to me (as far as coding it in the kernel is involved).

Thanks bye,

[Rusky]

If you want to migrate memory from one NUMA domain to another you'll need to copy the physical pages over (with the moral equivalent of memcpy). I'm not sure what Linux support for this looks like today, but initially they used the same mechanism as the swap system- the virtual addresses all stay the same but the page tables point to different physical pages.