028437445e
Due to Keystore's handling of sensitive information we cannot allow any other process ptrace privileges over Keystore. To silence SELinux denials llkd must ignore the keystore process. This CL adds keystore2 to the ignore list because it replaces keystore. In a followup the keystore entry will be removed. Bug: 170144267 Test: N/A Change-Id: I28eafc1f4341cdd371ad78d3a7a9ba8ac128c1b1
237 lines
9.8 KiB
Markdown
237 lines
9.8 KiB
Markdown
<!--
|
|
Project: /_project.yaml
|
|
Book: /_book.yaml
|
|
|
|
{% include "_versions.html" %}
|
|
-->
|
|
|
|
<!--
|
|
Copyright 2020 The Android Open Source Project
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
|
|
# Android Live-LocK Daemon (llkd)
|
|
|
|
Android 10 <!-- {{ androidQVersionNumber }} --> includes the Android Live-LocK Daemon
|
|
(`llkd`), which is designed to catch and mitigate kernel deadlocks. The `llkd`
|
|
component provides a default standalone implementation, but you can
|
|
alternatively integrate the `llkd` code into another service, either as part of
|
|
the main loop or as a separate thread.
|
|
|
|
## Detection scenarios <!-- {:#detection-scenarios} -->
|
|
|
|
The `llkd` has two detection scenarios: Persistent D or Z state, and persistent
|
|
stack signature.
|
|
|
|
### Persistent D or Z state <!-- {:#persistent-d-or-z-state} -->
|
|
|
|
If a thread is in D (uninterruptible sleep) or Z (zombie) state with no forward
|
|
progress for longer than `ro.llk.timeout_ms or ro.llk.[D|Z].timeout_ms`, the
|
|
`llkd` kills the process (or parent process). If a subsequent scan shows the
|
|
same process continues to exist, the `llkd` confirms a live-lock condition and
|
|
panics the kernel in a manner that provides the most detailed bug report for the
|
|
condition.
|
|
|
|
The `llkd` includes a self watchdog that alarms if `llkd` locks up; watchdog is
|
|
double the expected time to flow through the mainloop and sampling is every
|
|
`ro.llk_sample_ms`.
|
|
|
|
### Persistent stack signature <!-- {:#persistent-stack-signature} -->
|
|
|
|
For userdebug releases, the `llkd` can detect kernel live-locks using persistent
|
|
stack signature checking. If a thread in any state except Z has a persistent
|
|
listed `ro.llk.stack` kernel symbol that is reported for longer than
|
|
`ro.llk.timeout_ms` or `ro.llk.stack.timeout_ms`, the `llkd` kills the process
|
|
(even if there is forward scheduling progress). If a subsequent scan shows the
|
|
same process continues to exist, the `llkd` confirms a live-lock condition and
|
|
panics the kernel in a manner that provides the most detailed bug report for the
|
|
condition.
|
|
|
|
Note: Because forward scheduling progress is allowed, the `llkd` does not
|
|
perform [ABA detection](https://en.wikipedia.org/wiki/ABA_problem){:.external}.
|
|
|
|
The `lldk` check persists continuously when the live lock condition exists and
|
|
looks for the composed strings `" symbol+0x"` or `" symbol.cfi+0x"` in the
|
|
`/proc/pid/stack` file on Linux. The list of symbols is in `ro.llk.stack` and
|
|
defaults to the comma-separated list of
|
|
"`cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable`".
|
|
|
|
Symbols should be rare and short-lived enough that on a typical system the
|
|
function is seen only once in a sample over the timeout period of
|
|
`ro.llk.stack.timeout_ms` (samples occur every `ro.llk.check_ms`). Due to lack
|
|
of ABA protection, this is the only way to prevent a false trigger. The symbol
|
|
function must appear below the function calling the lock that could contend. If
|
|
the lock is below or in the symbol function, the symbol appears in all affected
|
|
processes, not just the one that caused the lockup.
|
|
|
|
## Coverage <!-- {:#coverage} -->
|
|
|
|
The default implementation of `llkd` does not monitor `init`, `[kthreadd]`, or
|
|
`[kthreadd]` spawns. For the `llkd` to cover `[kthreadd]`-spawned threads:
|
|
|
|
* Drivers must not remain in a persistent D state,
|
|
|
|
OR
|
|
|
|
* Drivers must have mechanisms to recover the thread should it be killed
|
|
externally. For example, use `wait_event_interruptible()` instead of
|
|
`wait_event()`.
|
|
|
|
If one of the above conditions is met, the `llkd` ignorelist can be adjusted to
|
|
cover kernel components. Stack symbol checking involves an additional process
|
|
ignore list to prevent sepolicy violations on services that block `ptrace`
|
|
operations.
|
|
|
|
## Android properties <!-- {:#android-properties} -->
|
|
|
|
The `llkd` responds to several Android properties (listed below).
|
|
|
|
* Properties named `prop_ms` are in milliseconds.
|
|
* Properties that use comma (,) separator for lists use a leading separator to
|
|
preserve the default entry, then add or subtract entries with optional plus
|
|
(+) and minus (-) prefixes respectively. For these lists, the string "false"
|
|
is synonymous with an empty list, and blank or missing entries resort to the
|
|
specified default value.
|
|
|
|
### ro.config.low_ram <!-- {:#ro-config-low-ram} -->
|
|
|
|
Device is configured with limited memory.
|
|
|
|
### ro.debuggable <!-- {:#ro-debuggable} -->
|
|
|
|
Device is configured for userdebug or eng build.
|
|
|
|
### ro.llk.sysrq_t <!-- {:#ro-llk-sysrq-t} -->
|
|
|
|
If property is "eng", the default is not `ro.config.low_ram` or `ro.debuggable`.
|
|
If true, dump all threads (`sysrq t`).
|
|
|
|
### ro.llk.enable <!-- {:#ro-llk-enable} -->
|
|
|
|
Allow live-lock daemon to be enabled. Default is false.
|
|
|
|
### llk.enable <!-- {:#llk-enable} -->
|
|
|
|
Evaluated for eng builds. Default is `ro.llk.enable`.
|
|
|
|
### ro.khungtask.enable <!-- {:#ro-khungtask-enable} -->
|
|
|
|
Allow `[khungtask]` daemon to be enabled. Default is false.
|
|
|
|
### khungtask.enable <!-- {:#khungtask-enable} -->
|
|
|
|
Evaluated for eng builds. Default is `ro.khungtask.enable`.
|
|
|
|
### ro.llk.mlockall <!-- {:#ro-llk-mlockall} -->
|
|
|
|
Enable call to `mlockall()`. Default is false.
|
|
|
|
### ro.khungtask.timeout <!-- {:#ro-khungtask-timeout} -->
|
|
|
|
`[khungtask]` maximum time limit. Default is 12 minutes.
|
|
|
|
### ro.llk.timeout_ms <!-- {:#ro-llk-timeout-ms} -->
|
|
|
|
D or Z maximum time limit. Default is 10 minutes. Double this value to set the
|
|
alarm watchdog for `llkd`.
|
|
|
|
### ro.llk.D.timeout_ms <!-- {:#ro-llk-D-timeout-ms} -->
|
|
|
|
D maximum time limit. Default is `ro.llk.timeout_ms`.
|
|
|
|
### ro.llk.Z.timeout_ms <!-- {:#ro-llk-Z-timeout-ms} -->
|
|
|
|
Z maximum time limit. Default is `ro.llk.timeout_ms`.
|
|
|
|
### ro.llk.stack.timeout_ms <!-- {:#ro-llk-stack-timeout-ms} -->
|
|
|
|
Checks for persistent stack symbols maximum time limit. Default is
|
|
`ro.llk.timeout_ms`. **Active only on userdebug or eng builds**.
|
|
|
|
### ro.llk.check_ms <!-- {:#ro-llk-check-ms} -->
|
|
|
|
Samples of threads for D or Z. Default is two minutes.
|
|
|
|
### ro.llk.stack <!-- {:#ro-llk-stack} -->
|
|
|
|
Checks for kernel stack symbols that if persistently present can indicate a
|
|
subsystem is locked up. Default is
|
|
`cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable`
|
|
comma-separated list of kernel symbols. The check doesn't do forward scheduling
|
|
ABA except by polling every `ro.llk_check_ms` over the period
|
|
`ro.llk.stack.timeout_ms`, so stack symbols should be exceptionally rare and
|
|
fleeting (it is highly unlikely for a symbol to show up persistently in all
|
|
samples of the stack). Checks for a match for `" symbol+0x"` or
|
|
`" symbol.cfi+0x"` in stack expansion. **Available only on userdebug or eng
|
|
builds**; security concerns on user builds result in limited privileges that
|
|
prevent this check.
|
|
|
|
### ro.llk.ignorelist.process <!-- {:#ro-llk-ignorelist-process} -->
|
|
|
|
The `llkd` does not watch the specified processes. Default is `0,1,2` (`kernel`,
|
|
`init`, and `[kthreadd]`) plus process names
|
|
`init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd, [watchdogd],[watchdogd/0],...,[watchdogd/get_nprocs-1]`.
|
|
A process can be a `comm`, `cmdline`, or `pid` reference. An automated default
|
|
can be larger than the current maximum property size of 92.
|
|
|
|
Note: `false` is an extremely unlikely process to want to ignore.
|
|
|
|
### ro.llk.ignorelist.parent <!-- {:#ro-llk-ignorelist-parent} -->
|
|
|
|
The `llkd` does not watch processes that have the specified parent(s). Default
|
|
is `0,2,adbd&[setsid]` (`kernel`, `[kthreadd]`, and `adbd` only for zombie
|
|
`setsid`). An ampersand (&) separator specifies that the parent is ignored only
|
|
in combination with the target child process. Ampersand was selected because it
|
|
is never part of a process name; however, a `setprop` in the shell requires the
|
|
ampersand to be escaped or quoted, although the `init rc` file where this is
|
|
normally specified does not have this issue. A parent or target process can be a
|
|
`comm`, `cmdline`, or `pid` reference.
|
|
|
|
### ro.llk.ignorelist.uid <!-- {:#ro-llk-ignorelist-uid} -->
|
|
|
|
The `llkd` does not watch processes that match the specified uid(s).
|
|
Comma-separated list of uid numbers or names. Default is empty or false.
|
|
|
|
### ro.llk.ignorelist.process.stack <!-- {:#ro-llk-ignorelist-process-stack} -->
|
|
|
|
The `llkd` does not monitor the specified subset of processes for live lock stack
|
|
signatures. Default is process names
|
|
`init,lmkd.llkd,llkd,keystore,keystore2,ueventd,apexd,logd`. Prevents the sepolicy
|
|
violation associated with processes that block `ptrace` (as these can't be
|
|
checked). **Active only on userdebug and eng builds**. For details on build
|
|
types, refer to [Building Android](/setup/build/building#choose-a-target).
|
|
|
|
## Architectural concerns <!-- {:#architectural-concerns} -->
|
|
|
|
* Properties are limited to 92 characters. However, this is not limited for
|
|
defaults defined in the `include/llkd.h` file in the sources.
|
|
* The built-in `[khungtask]` daemon is too generic and trips on driver code that
|
|
sits around in D state too much. Switching drivers to sleep, or S state,
|
|
would make task(s) killable, and need to be resurrectable by drivers on an
|
|
as-need basis.
|
|
|
|
## Library interface (optional) <!-- {:#library-interface-optional} -->
|
|
|
|
You can optionally incorporate the `llkd` into another privileged daemon using
|
|
the following C interface from the `libllkd` component:
|
|
|
|
```
|
|
#include "llkd.h"
|
|
bool llkInit(const char* threadname) /* return true if enabled */
|
|
unsigned llkCheckMillseconds(void) /* ms to sleep for next check */
|
|
```
|
|
|
|
If a threadname is provided, a thread automatically spawns, otherwise the caller
|
|
must call `llkCheckMilliseconds` in its main loop. The function returns the
|
|
period of time before the next expected call to this handler.
|