Description
In the Linux kernel, the following vulnerability has been resolved:
drm/v3d: Serialize the scheduler timeout handlers
V3D exposes several independent hardware queues (BIN, RENDER, TFU and
CSD) but has only a single, global reset. A timeout on any one queue
therefore has to stop, reset and restart the schedulers of every other
queue as well. That makes concurrent timeout handlers unsafe.
`reset_lock` was never able to make them safe, as a driver-side lock can
only cover the driver's &drm_sched_backend_ops.timedout_job callback.
The scheduler handles the timed out job and its pending list around that
callback, outside of the driver's control, so a global reset triggered
by one queue can still interfere with another queue that is in the
middle of handling a timeout of its own.
Consequently, if a reset happens in the CSD queue while a CL-intensive
application is running, the global reset stops and restarts the CL
queue's scheduler while that queue is handling a timeout of its own. As
drm_sched_stop() and drm_sched_start() subtract and add the credits of
every job sitting on the pending list of the scheduler they are called
on, and as the CL queue's handler concurrently takes its job off that
same list and puts it back, the stop and the start no longer see the
same set of jobs. The CL queue is left with more credits in flight than
its limit:
[ 327.302739] ------------[ cut here ]------------
[ 327.302744] WARNING: CPU: 2 PID: 43 at drivers/gpu/drm/scheduler/sched_main.c:102 drm_sched_run_job_work+0x238/0x4d0 [gpu_sched]
[ 327.302884] CPU: 2 UID: 0 PID: 43 Comm: kworker/u16:1 Not tainted 6.18.39-v8-16k+ #3 PREEMPT
[ 327.302889] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[ 327.302893] Workqueue: v3d_bin drm_sched_run_job_work [gpu_sched]
[ 327.302984] Call trace:
[ 327.302987] drm_sched_run_job_work+0x238/0x4d0 [gpu_sched] (P)
[ 327.302997] process_scheduled_works+0x180/0x3d0
[ 327.303010] worker_thread+0x268/0x3e8
[ 327.303016] kthread+0x140/0x250
[ 327.303022] ret_from_fork+0x10/0x20
[ 327.303031] ---[ end trace 0000000000000000 ]---
From that point on, the credit count of the CL queue is broken, causing
a complete GPU hang and UI freeze.
The DRM scheduler already provides a mechanism to serialize the timeout
handlers of different schedulers: an ordered workqueue passed as
drm_sched_init()'s @timeout_wq parameter. By default, each scheduler
queues its timeout work on the system workqueue, which runs the handlers
concurrently. Give all of the queues a shared ordered workqueue instead,
as recommended by the DRM scheduler documentation for hardware that has
distinct queues but resets globally.
Références
