android_system_core/logd
Tom Cherry f74503dd46 logd: optionally track the full size of log buffers
ChattyLogBuffer ignores the metadata (timestamp, pid, std::list<>
iterators, etc) of log entries when calculating the size used by a
given log buffer. For example, if 1MB is the specified size of the
'main' log buffer, logd will use between ~1.3MB and ~2MB of overall
memory for 'main' log buffer.  LogStatistics does track the overall
memory used and labels it 'Overhead', however this 'Overhead' is only
informative and is not used for Pruning or Chatty calculations.

This is problematic, since it makes logd's memory usage inconsistent:
depending on the pattern of logging, there can be substantially more
memory used than the specified log buffer size.  This is further
complicated by the fact that chatty messages are entirely metadata and
therefore not counted as contributing to the log buffer size.

This change would switch logd to always track the full size of log
buffers, but there are two problems with this approach:
1) Unless users double their buffer sizes, then they'd have
   substantially fewer logs after the change
2) Chatty logic would change and it's difficult to evaluate.

Therefore this change only provides the framework to track the full
size of log buffers.  This allows an apples to apples comparison of
ChattyLogBuffer and SerializedLogBuffer.  With this option enabled,
logd reports the following values:

ChattyLogBuffer:
Total log size (logcat -g), 'Total' / 'Now' (logcat -S), and
'Overhead' (logcat -S) all report the full size of log entries
including metadata.

SerializedLogBuffer:
Total log size (logcat -g) and 'Overhead' (logcat -S) report the
compressed size of the log entries including metadata.
'Total' / 'Now' (logcat -S) reports the uncompressed size of the log
entries that are available including metadata.

Test: logging statistics are correct
Change-Id: If17682af8bb605f31387d7b210b69a301dd48f07
2020-07-01 14:35:33 -07:00
..
fuzz logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
.clang-format logd: use .clang-format-4 like the rest of system/core 2019-01-16 15:20:15 -08:00
Android.bp logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
AndroidTest.xml logd: add LogBufferTest.cpp 2020-05-18 15:51:08 -07:00
auditctl.cpp introduce auditctl and use it to configure SELinux throttling 2019-04-09 13:19:08 -07:00
ChattyLogBuffer.cpp logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
ChattyLogBuffer.h logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
ChattyLogBufferTest.cpp logd: move leading_dropped logic into FlushTo() 2020-06-02 13:26:48 -07:00
CommandListener.cpp logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
CommandListener.h logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
CompressionEngine.cpp logd: replace std::vector<uint8_t> in SerializedLogChunk 2020-06-24 16:19:28 -07:00
CompressionEngine.h logd: replace std::vector<uint8_t> in SerializedLogChunk 2020-06-24 16:19:28 -07:00
event.logtags liblog: logprint supports number of seconds time event field 2017-04-14 12:54:25 -07:00
libaudit.cpp logd: make liblogd just the core logd functionality 2020-05-18 13:03:48 -07:00
libaudit.h logd: make liblogd just the core logd functionality 2020-05-18 13:03:48 -07:00
LogAudit.cpp logd: make LogBuffer an interface 2020-05-12 15:39:19 -07:00
LogAudit.h logd: refactor LastLogTimes a bit 2020-05-12 15:39:11 -07:00
LogBuffer.h logd: move leading_dropped logic into FlushTo() 2020-06-02 13:26:48 -07:00
LogBufferElement.cpp logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
LogBufferElement.h logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00
LogBufferTest.cpp logd: add a SerializedLogBuffer suitable for compression 2020-06-12 14:35:30 -07:00
LogBufferTest.h logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
logd.rc logd: don't drop user/groups/capabilities/priority in drop_privs() 2019-06-07 15:52:27 +00:00
logd_test.cpp logd: add LogBufferTest.cpp 2020-05-18 15:51:08 -07:00
LogKlog.cpp logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
LogKlog.h logd: remove SocketClient from LogBuffer and LogBufferElement 2020-05-14 19:53:45 -07:00
LogListener.cpp logd: remove class LogCommand, rename LogCommand.cpp to LogPermissions.cpp 2020-05-15 11:58:47 -07:00
LogListener.h logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00
LogPermissions.cpp logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00
LogPermissions.h logd: remove class LogCommand, rename LogCommand.cpp to LogPermissions.cpp 2020-05-15 11:58:47 -07:00
LogReader.cpp logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00
LogReader.h logd: remove SocketClient from LogBuffer and LogBufferElement 2020-05-14 19:53:45 -07:00
LogReaderList.cpp logd: create FlushToState class 2020-06-01 14:45:02 -07:00
LogReaderList.h logd: create FlushToState class 2020-06-01 14:45:02 -07:00
LogReaderThread.cpp logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00
LogReaderThread.h logd: add a SerializedLogBuffer suitable for compression 2020-06-12 14:35:30 -07:00
LogStatistics.cpp logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
LogStatistics.h logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
logtagd.rc Relax permissions on /data/misc/logd 2020-01-21 18:39:19 -08:00
LogTags.cpp logd: add fuzzer for SerializedLogBuffer and other improvements 2020-06-23 11:37:49 -07:00
LogTags.h logd: create SimpleLogBuffer and implement ChattyLogBuffer in terms of it 2020-05-21 14:23:45 -07:00
LogUtils.h logd: use libbase logging 2020-06-03 16:29:48 -07:00
LogWriter.h logd: drop can_read_security_logs 2020-06-01 14:45:02 -07:00
main.cpp logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
OWNERS Add OWNERS. 2017-12-07 13:30:03 -08:00
PruneList.cpp logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
PruneList.h logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
README.auditd
README.property logging: Use more inclusive language. 2020-06-16 14:31:31 -07:00
rwlock.h logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00
SerializedData.h logd: replace std::vector<uint8_t> in SerializedLogChunk 2020-06-24 16:19:28 -07:00
SerializedFlushToState.cpp logd: fix use after resize of contents_ vector 2020-06-24 15:31:46 -07:00
SerializedFlushToState.h logd: fix use after resize of contents_ vector 2020-06-24 15:31:46 -07:00
SerializedFlushToStateTest.cpp logd: fix use after resize of contents_ vector 2020-06-24 15:31:46 -07:00
SerializedLogBuffer.cpp logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
SerializedLogBuffer.h logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
SerializedLogChunk.cpp logd: replace std::vector<uint8_t> in SerializedLogChunk 2020-06-24 16:19:28 -07:00
SerializedLogChunk.h logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
SerializedLogChunkTest.cpp logd: add a SerializedLogBuffer suitable for compression 2020-06-12 14:35:30 -07:00
SerializedLogEntry.h logd: optionally track the full size of log buffers 2020-07-01 14:35:33 -07:00
SimpleLogBuffer.cpp logd: simplify Clear() + Prune() logic 2020-06-11 07:59:01 -07:00
SimpleLogBuffer.h logd: fix various clang-tidy issues 2020-06-17 11:40:55 -07:00

The properties that logd and friends react to are:

name                       type default  description
ro.logd.auditd             bool   true   Enable selinux audit daemon
ro.logd.auditd.dmesg       bool   true   selinux audit messages sent to dmesg.
ro.logd.auditd.main        bool   true   selinux audit messages sent to main.
ro.logd.auditd.events      bool   true   selinux audit messages sent to events.
persist.logd.security      bool   false  Enable security buffer.
ro.organization_owned      bool   false  Override persist.logd.security to false
ro.logd.kernel             bool+ svelte+ Enable klogd daemon
ro.logd.statistics         bool+ svelte+ Enable logcat -S statistics.
ro.debuggable              number        if not "1", logd.statistics &
                                         ro.logd.kernel default false.
logd.logpersistd.enable    bool   auto   Safe to start logpersist daemon service
logd.logpersistd          string persist Enable logpersist daemon, "logcatd"
                                         turns on logcat -f in logd context.
					 Responds to logcatd, clear and stop.
logd.logpersistd.buffer          persist logpersistd buffers to collect
logd.logpersistd.size            persist logpersistd size in MB
logd.logpersistd.rotate_kbytes   	 persist logpersistd outout file size in KB.
persist.logd.logpersistd   string        Enable logpersist daemon, "logcatd"
                                         turns on logcat -f in logd context.
persist.logd.logpersistd.buffer    all   logpersistd buffers to collect
persist.logd.logpersistd.size      256   logpersistd size in MB
persist.logd.logpersistd.count     256   sets max number of rotated logs to <count>.
persist.logd.logpersistd.rotate_kbytes   1024  logpersistd output file size in KB
persist.logd.size          number  ro    Global default size of the buffer for
                                         all log ids at initial startup, at
                                         runtime use: logcat -b all -G <value>
ro.logd.size               number svelte default for persist.logd.size. Larger
                                         platform default sizes than 256KB are
                                         known to not scale well under log spam
                                         pressure. Address the spam first,
                                         resist increasing the log buffer.
persist.logd.size.<buffer> number  ro    Size of the buffer for <buffer> log
ro.logd.size.<buffer>      number svelte default for persist.logd.size.<buffer>
ro.config.low_ram          bool   false  if true, logd.statistics,
                                         ro.logd.kernel default false,
                                         logd.size 64K instead of 256K.
persist.logd.filter        string        Pruning filter to optimize content.
                                         At runtime use: logcat -P "<string>"
ro.logd.filter       string "~! ~1000/!" default for persist.logd.filter.
                                         This default means to prune the
                                         oldest entries of chattiest UID, and
                                         the chattiest PID of system
                                         (1000, or AID_SYSTEM).
log.tag                   string persist The global logging level, VERBOSE,
                                         DEBUG, INFO, WARN, ERROR, ASSERT or
                                         SILENT. Only the first character is
                                         the key character.
persist.log.tag            string build  default for log.tag
log.tag.<tag>             string persist The <tag> specific logging level.
persist.log.tag.<tag>      string build  default for log.tag.<tag>

NB:
- auto - managed by /init
- bool+ - "true", "false" and comma separated list of "eng" (forced false if
  ro.debuggable is not "1") or "svelte" (forced false if ro.config.low_ram is
  true).
- svelte - see ro.config.low_ram for details.
- svelte+ - see ro.config.low_ram and ro.debuggable for details.
- ro - <base property> temporary override, ro.<base property> platform default.
- persist - <base property> override, persist.<base property> platform default.
- build - VERBOSE for native, DEBUG for jvm isLoggable, or developer option.
- number - support multipliers (K or M) for convenience. Range is limited
  to between 64K and 256M for log buffer sizes. Individual log buffer ids
  such as main, system, ... override global default.
- Pruning filter rules are specified as UID, UID/PID or /PID. A '~' prefix indicates that elements
  matching the rule should be pruned with higher priority otherwise they're pruned with lower
  priority. All other pruning activity is oldest first. Special case ~! represents an automatic
  pruning for the noisiest UID as determined by the current statistics.  Special case ~1000/!
  represents pruning of the worst PID within AID_SYSTEM when AID_SYSTEM is the noisiest UID.