up_squared: A lot of fails in kernel and libraries components

[arikgreen]

Describe the bug

Since March, 08 we observed a lot of the new fails on platform: up_squared/apollo_lake
List of fail tests:

kernel.common.profiling					FAILED Timeout (device: up2, 123.277s <zephyr>)
kernel.ipi_optimize.smp					FAILED Timeout (device: up2, 123.652s <zephyr>)
kernel.threads.thread_stack				FAILED Timeout (device: up2, 123.014s <zephyr>)
kernel.multiprocessing.smp.affinity			FAILED Timeout (device: up2, 242.988s <zephyr>)
kernel.memory_protection				FAILED Timeout (device: up2, 122.961s <zephyr>)
kernel.multiprocessing.smp.minimallibc			FAILED Timeout (device: up2, 242.961s <zephyr>)
kernel.multiprocessing.smp.affinity.custom_rom_offset	FAILED Timeout (device: up2, 243.041s <zephyr>)
kernel.multiprocessing.smp				FAILED Timeout (device: up2, 243.313s <zephyr>)
libraries.libc.common.picolibc				FAILED Timeout (device: up2, 123.122s <zephyr>)
libraries.libc.common.newlib_nano			FAILED Timeout (device: up2, 122.960s <zephyr>)
libraries.libc.common.picolibc.module			FAILED Timeout (device: up2, 123.041s <zephyr>)
libraries.libc.common					FAILED Timeout (device: up2, 123.146s <zephyr>)
libraries.libc.common.newlib				FAILED Timeout (device: up2, 123.424s <zephyr>)
libraries.libc.common.minimal				FAILED Timeout (device: up2, 122.981s <zephyr>)
kernel.ipi_cascade.smp					FAILED Timeout (device: up2, 123.392s <zephyr>)

twister.log

IMO it is a regression.
Specific commit since the problem : 427f2c6

To Reproduce
run twister for the platform up_squared/apollo_lake and above test scope

west twister -p up_squared/apollo_lake -s kernel.multiprocessing.smp.affinity.custom_rom_offset

Expected behavior

test suite passed for above tests

Impact

This is a showstopper.

Logs and console output

INFO - 7 of 22 executed test configurations passed (31.82%), 0 built (not run), 15 failed, 0 errored, with no warnings in 3886.15 seconds.

Environment (please complete the following information):

• OS: Linux Ubuntu 22.04.5 LTS
• twister - INFO - Using Ninja..
• twister - INFO - Zephyr version: v4.1.0-227-g427f2c60da0b
• twister - INFO - Using 'zephyr' toolchain.
• Commit: 427f2c60da0beb8b68e85bbcf2cbbfd59ad469f8

Additional context

[peter-mitsis]

I must have messed something up when attempting the direct IPI support for x86. For the moment, I suggest reverting the offending commit.

[peter-mitsis]

Did some more digging, on QEMU (where this was tested), CONFIG_X2APIC is not set. However, on upsquared/apollo_lake, CONFIG_X2APIC is enabled. This is significant as the behavior of the routine z_loapic_ipi() differs based on the enablement/disablement of this Kconfig option.

With this knowledge, we may still be able to keep the directed IPI support for x86 only when CONFIG_X2APIC is not enabled to get us past this stopper, and then work on fixing the support for it when CONFIG_X2APIC is enabled.

[arikgreen]

I attached the twister log

[nashif]

Logs and console output

INFO - 7 of 22 executed test configurations passed (31.82%), 0 built (not run), 15 failed, 0 errored, with no warnings in 3886.15 seconds.

in the future, please do not put twister output in the log, the expectation is to have the console output from the device when running the tests, so this is more appropriate for example and can be helpful:

*** Booting Zephyr OS build 4.1.99 (delayed boot 500ms) ***
Running TESTSUITE ipi
===================================================================
START - test_arch_sched_broadcast_ipi
 PASS - test_arch_sched_broadcast_ipi in 0.038 seconds
===================================================================
START - test_arch_sched_directed_ipi

    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/ipi_optimize/src/main.c:231: ipi_test_arch_sched_directed_ipi: (set[j] == 1 is false)
Direct-Expected 1, got 0