[SOLVED]Multicore task scheduling issue

[shizi]

I'm working on an x86_64 multicore operating system. I already have memory management, interrupts, and a time framework in place. The idle task prints every 50 times it is scheduled. I'm currently stuck on task management:

The BSP can trigger timer interrupts normally, and idle keeps running (the log shows `[APIC] timer IRQ` and idle counter output).
The AP never triggers a timer interrupt; not a single `[APIC] timer IRQ` appears.
I tried adding several kernel threads, noticed that PID allocation is abnormal, and the system seems to only run the idle thread – other threads are never scheduled. Through debugging I confirmed that the test threads are enqueued (`rr_enqueue` is called), but they never get scheduled.

The initialization flow is as follows.

The BSP first executes `task_data_init` (only once):

Code: Select all

bool task_data_init(void) {
    uint32_t max_cpu = bootboot->numcores;
    uint64_t per_cpu_task_stopped_size = sizeof(uint32_t) + (sizeof(task_state_struct) * max_cpu); 
    per_cpu_task_state_ptr = kheap_alloc(per_cpu_task_stopped_size);
    if (!per_cpu_task_state_ptr) return false;

    for (int i = 0; i < max_cpu; i++) INIT_LIST_HEAD(&per_cpu_task_state_ptr->state[i].stopped);

    id_map = dynarr_create(sizeof(struct task_struct *), TASK_ID_MAX);
    if (!id_map) return false;

    sched_func();

    idle = task_create_idle();

    TASK_PRINT("task data init success\n");

    return true;
}

Here the scheduler is registered (`sched_func`) and the idle task is created. In my own debug code, I create the test threads after `task_data_init` but before `smp_init` (using `task_create_kernel_thread`, each prints characters like A, B, etc.), and I confirmed they all call `rr_enqueue` and are added to the BSP's run queue.

Then all CPUs enter `task_init`:

Code: Select all

void task_init(void) {
    if (!idle) goto error; 
    smp_set_idle(idle);

    if (sched_class_ptr && sched_class_ptr->init) sched_class_ptr->init();

    clockevent_handle_t clockevent = clockevent_get(NULL);
    if (!clockevent) goto error;

    smp_set_clockevent(clockevent);
    clockevent_set_handler(clockevent, task_clock_event_handle);
    clockevent_set_mode(clockevent, CLOCKEVENT_MODE_ONESHOT);

    TASK_PRINT("task init success\n");

    task_set_next_timer();
    task_boot_sched();

    error:
        TASK_PANIC("system error");
}

All CPUs finish without panic, and the APs also successfully jump (they never return from `task_boot_sched`), but the APs never get any timer interrupts.

Log output:

Code: Select all

[BOOTKERNEL] BP CPU logical_id : 0
[BOOTKERNEL] BP CPU APIC_ID : 0
[KERNEL]ShiziOS KERNEL v0.1.0
...
[CLOCKEVENT] register clockevent : [“name” = "apic", “hz” = "100000000"]
...
[ACPI] ACPI init succeed
[IOAPIC] ioapic init success
[CLOCKEVENT] register clockevent : [“name” = "pit", “hz” = "1193182"]
[TASK]task data init success
[SMP] smp init succeed
[SMP] smp init succeed
[TASK]task init success
[IDLE] CPU 1
[TASK]task init success
[IDLE] CPU 0
[APIC] timer IRQ
[APIC] timer IRQ
…… (intermediate interrupt prints omitted)
[APIC] timer IRQ
[IDLE] CPU 0
[IDLE] count : 50
[APIC] timer IRQ
……

Later, I never see any AP output. Even when the BSP's counter reaches 350, there is still no AP output. This indicates that the AP is not receiving timer interrupts, even though in my code I set up the interrupt for each core.

Here is the scheduler enqueue and pick_next code:

Code: Select all

void rr_enqueue(struct task_struct *task) {
    per_cpu_sched *pcpu_sched = smp_get_sched();
    list_add_tail(&task->sched.list, &pcpu_sched->run_queue);
}

struct task_struct *rr_pick_next(void) {
    per_cpu_sched *pcpu_sched = smp_get_sched();

    if (list_empty(&pcpu_sched->run_queue)) return smp_get_idle();

    struct list_head *head = pcpu_sched->run_queue.next;
    list_move_tail(head, &pcpu_sched->run_queue);
    return list_entry(head, task_struct, sched.list);
}

My main goal is to solve the multithread scheduling and interrupt issues; I have no idea where to go next.

Since the code is quite large, I'm not pasting everything here. The main parts are in `kernel/task`. Here is the repository link:

https://github.com/shizi297/ShiziOS

[Octocontrabass]

What does your debugger say the APs are doing instead of receiving interrupts?

What does QEMU's "info lapic" say about the state of the local APICs on the APs?

[shizi]

What does your debugger say the APs are doing instead of receiving interrupts?

Thank you for your guidance!
I had never used the thread command in GDB before, which led me to believe that the AP had also entered idle. When you suggested that I check what the AP was doing, I specifically switched to the AP thread and looked. I discovered it was deadlocked because I was acquiring a lock in interrupt context. Now I no longer acquire that lock there (I removed the serial calls that were taking the lock), and it works fine now. Thanks again!