watchdog
对手机系统而言，因为肩负着接听电话和接收短信的“重任”，所以被寄予7x24小 时正常工作的希望。但是作为一个在嵌入式设备上运行的操作系统，Android运行中必须面对各种软硬件干扰，从最简单的代码出现死锁或者被阻塞，到内存越界导致的内存破坏，或者由于硬件问题导致的内存反转，甚至是极端工作环境下出现的cpu电子迁移和存储器消磁。这一切问题都可能导致系统服务发生难以预料的崩溃和死机。想解决这一问题，可以从正反两个方向出发，其一是提高软硬件在极端状态下的可靠性，如进行程序终止性验证，或选用抗辐射加固器件。但是基于成本考虑，普通的手机系统很难做到完全不出故障；另一个方法是及时发现系统崩溃并重启系统。手机系统的大部分的故障都会在重启后消失，不会影响继续使用。所以简单的办法是，如果检测到系统不正常了，将设备重新启动，这样用户就能继续使用了。那么如何才能判断系统是否正常呢。在早期的手机平台上通常的做法是在设备中增加一个硬件看门狗，软件系统必须定 时的向看门狗硬件中写值来表示自己没出故障（俗称“喂狗”），否则超过了规定的时间看门狗就会重新启动设备。
硬件看门狗的问题是它的功能比较单一，只能监控整个系统。早期的手机操作系统大多是单任务的，硬件看门狗勉强能胜任。Android的SystemServer是一个非常复杂的进程，里面运行的服务超过五十种，是最可能出问题的进程，因此有必要对SystemServer中运行的各种线程实施监控。但是如果使用硬件看门狗的工作方式，每个线程隔一段时间去喂狗，不但非常浪费CPU，而且会导致程序设计更加复杂。因此Android开发了watchdog类作为软件看门狗来监控SystemServer中的线程。一旦发现问题，WatchDog会杀死SystemServer进程。
SystemServer的父进程Zygote接收到SystemServer的死亡信号后，会杀死自己。Zygote进程死亡的信号传递到Init进程后，Init进程会杀死Zygote进程所有的子进程并重启Zygote。这样整个手机相当于重启一遍。通常SystemServer出现问题和kernel并没有关系，所以这种“软重启”大部分时候都能够解决问题。而且这种“软重启”的速度更快，对用户的影响也更小。
WatchDog是在SystemServer进程中被初始化和启动的。在SystemServer 的run方法中，各种Android服务被注册和启动，其中也包括了WatchDog的初始化和启动。代码如下：
final Watchdog watchdog = Watchdog.getinstance();
watchdog.init(context, mactivitymanagerService); Watchdog.getInstance().start();前面说到WatchDog是在SystemServer.java中通过getInstance方法创建的，其具体实现方式如下：
public static Watchdog getInstance() {
if (sWatchdog == null) {
sWatchdog = new Watchdog(); //单例模式创建实例
}
return sWatchdog;
}
private Watchdog() {
super("watchdog");
// Initialize handler checkers for each common thread we want to check. Note
// that we are not currently checking the background thread, since it can
// potentially hold longer running operations with no guarantees about the timeliness
// of operations there.
// The shared foreground thread is the main checker. It is where we
// will also dispatch monitor checks and do other work.
mMonitorChecker = new HandlerChecker(FgThread.getHandler(),
"foreground thread", DEFAULT_TIMEOUT);
mHandlerCheckers.add(mMonitorChecker);
// Add checker for main thread. We only do a quick check since there
// can be UI running on the thread.
mHandlerCheckers.add(new HandlerChecker(new Handler(Looper.getMainLooper()),
"main thread", DEFAULT_TIMEOUT));
// Add checker for shared UI thread.
mHandlerCheckers.add(new HandlerChecker(UiThread.getHandler(),
"ui thread", DEFAULT_TIMEOUT));
// And also check IO thread.
mHandlerCheckers.add(new HandlerChecker(IoThread.getHandler(),
"i/o thread", DEFAULT_TIMEOUT));
// And the display thread.
mHandlerCheckers.add(new HandlerChecker(DisplayThread.getHandler(),
"display thread", DEFAULT_TIMEOUT));
// Initialize monitor for Binder threads.
addMonitor(new BinderThreadMonitor());
} public void addMonitor(Monitor monitor) {
synchronized (this) {
if (isAlive()) {
throw new runtimeexception("Monitors can't be added once the Watchdog is running");
}
mMonitorChecker.addMonitor(monitor);
}
}以ActivityManagerService.java为例，为向watchdog注册monitor()回调，首先需要继承watchdog.Monitor接口：
public class ActivityManagerService extends ActivityManagerNativeEx
implements Watchdog.Monitor, BATteryStatsImpl.BatteryCallback { Watchdog.getInstance().addMonitor(this);
Watchdog.getInstance().addThread(mHandler); /** In this method we try to acquire our lock to make sure that we have not deadlocked */
public void monitor() {
synchronized (this) { }
} public void addThread(Handler thread) {
addThread(thread, DEFAULT_TIMEOUT);
}
public void addThread(Handler thread, long timeoutMillis) {
synchronized (this) {
if (isAlive()) {
throw new RuntimeException("Threads can't be added once the Watchdog is running");
}
final String name = thread.getLooper().getThread().getName();
mHandlerCheckers.add(new HandlerChecker(thread, name, timeoutMillis));
}
} /**
* Used for checking status of handle threads and scheduling monitor callbacks.
*/
public final class HandlerChecker implements Runnable {
private final Handler mHandler;
private final String mName;
private final long mWaitMax;
private final ArrayList<Monitor> mMonitors = new ArrayList<Monitor>();
private boolean mCompleted;
private Monitor mCurrentMonitor;
private long mStartTime;
HandlerChecker(Handler handler, String name, long waitMaxMillis) {
mHandler = handler;
mName = name;
mWaitMax = waitMaxMillis;
mCompleted = true;
}之所以不能在watchdog的主线程中回调monitor()方法，是由于如果被监控服务的关键区被占用，其monitor()方法可能需要一段时间才能返回。这样就无法保证watchdog每次个检测周期都是30s，所以必须交由foreground thread代为检查。
addMonitor()中会把每个monitor添加到mMonitorChecker也就是foreground thread的HandlerChecker中。除了它以外，所有HandlerChecker的mMonitors都是空的。
当watchdog的主循环开始运行后，每隔30秒，都会依次调用所有HandlerChecker的scheduleCheckLocked()方法。对于foreground thread的HandlerChecker，由于它的mMonitors不为空，需要它去锁各服务的monitor()来检查是否出现死锁，因此每个检测周期都要执行它。
对于其他的HandlerChecker，需要判断线程的Looper是否处于Idling，若为空就说明前一个消息已经执行完毕正在等下一个，消息循环肯定没阻塞，不用继续检测直接跳过本轮。
如果线程的消息循环不是Idling状态，说明服务的主线程正在处理某个消息，有阻塞的可能，就需要使用PostAtFrontOfqueue发出消息到消息队列，并记录下当前系统时间，同时将mComplete置为false，标明已经发出一个消息正在等待处理。
public void scheduleCheckLocked() {
if (mMonitors.size() == 0 && mHandler.getLooper().getQueue().isPolling()) {
// If the target looper has recently been polling, then
// there is no reason to enqueue our checker on it since that
// is as good as it not being deadlocked. This avoid having
// to do a context switch to check the thread. Note that we
// only do this if mCheckreboot is false and we have no
// monitors, since those would need to be executed at this point.
mCompleted = true;
return;
}
if (!mCompleted) {
// we already have a check in flight, so no need
return;
}
mCompleted = false;
mCurrentMonitor = null;
mStartTime = SystemClock.uptimeMillis();
mHandler.postAtFrontOfQueue(this);
} @Override
public void run() {
final int size = mMonitors.size();
for (int i = 0 ; i < size ; i++) {
synchronized (Watchdog.this) {
mCurrentMonitor = mMonitors.get(i);
}
mCurrentMonitor.monitor();
}
synchronized (Watchdog.this) {
mCompleted = true;
mCurrentMonitor = null;
}
}
} public int getCompletionStateLocked() {
if (mCompleted) {
return COMPLETED;
} else {
long latency = SystemClock.uptimeMillis() - mStartTime;
if (latency < mWaitMax/2) {
return WAITING;
} else if (latency < mWaitMax) {
return WAITED_HALF;
}
}
return OVERDUE;
}SystemServer调用watchdog的start方法，watchdog便开始在自己线程的while循环中运行，以达到每30s检测一次的目的：
@Override
public void run() {
boolean waitedHalf = false;
while (true) {
final ArrayList<HandlerChecker> blockedCheckers;
final String subject;
final boolean allowRestart;
int debuggerWasConnected = 0;
synchronized (this) {
long timeout = CHECK_Interval;
// Make sure we (re)spin the checkers that have become idle within
// this wait-and-check interval
for (int i=0; i<mHandlerCheckers.size(); i++) { //遍历各个HandlerChecker，依次检查前台，ui，主线程等系统主要线程
HandlerChecker hc = mHandlerCheckers.get(i);
hc.scheduleCheckLocked();
}检查主线程是否阻塞的方法是，如果线程Looper状态不是Idling，就通过HandlerChecker的postAtFrontOfQueue方法发送一个消息。稍后检测这个消息是否超时未返回。
通过postAtFrontOfQueue送出消息后睡眠30s。注意这里使用uptimeMillis()计算时间，不计手机在睡眠中度过的时间。这是由于手机睡眠时系统服务同样也在睡眠，无法响应watchdog送出的消息，如果把睡眠时间计算在内当手机被再次唤醒时会导致watchdog认为时间已经过去了很久，从而发生误杀。
// NOTE: We use uptimeMillis() here because we do not want to increment the time we
// wait while asleep. If the device is asleep then the thing that we are waiting
// to timeout on is asleep as well and won't have a chance to run, causing a false
// positive on when to kill things.
long start = SystemClock.uptimeMillis(); //使用uptimeMills不把手机睡眠时间算进入，手机睡眠时系统服务同样睡眠，状态无法响应watchdog会导致误杀
while (timeout > 0) {
if (Debug.isDebuggerConnected()) {
debuggerWasConnected = 2;
}
try {
wait(timeout);
} catch (InterruptedException e) {
Log.wtf(TAG, e);
}
if (Debug.isDebuggerConnected()) {
debuggerWasConnected = 2;
} //CHECK_INTERVAL的默认时间是30s，此为第一次等待时间，WatchDog判断对象是否死锁的最长等待时间为1min
timeout = CHECK_INTERVAL - (SystemClock.uptimeMillis() - start);
} final int waitState = evaluateCheckerCompletionLocked();
if (waitState == COMPLETED) {
// The monitors have returned; reset
waitedHalf = false; //所有服务都正常，reset
continue;
} else if (waitState == WAITING) {
// still waiting but within their configured intervals; back off and recheck
continue;
} else if (waitState == WAITED_HALF) {
if (!waitedHalf) {
// We've waited half the deadlock-detection interval. Pull a stack
// trace and wait another half.
ArrayList<integer> pids = new ArrayList<Integer>();
pids.add(Process.myPid());
ActivityManagerService.dumpStackTraces(true, pids, null, null,
NATIVE_STACKS_OF_INTEREST);
waitedHalf = true;
}
continue;
} // something is overdue!
blockedCheckers = getBlockedCheckersLocked(); //WatchDog超时，获取那个服务超时阻塞，生成崩溃描述符
subject = describeCheckersLocked(blockedCheckers); //判断是否重启
allowRestart = mAllowRestart;
}
// If we got here, that means that the system is most likely hung.
// First collect stack traces from all threads of the system process.
// Then kill this process so that the system will restart.
EventLog.writeEvent(EventLogTags.WATCHDOG, subject);
ArrayList<Integer> pids = new ArrayList<Integer>();
pids.add(Process.myPid());
if (mPhonePid > 0) pids.add(mPhonePid);
// Pass !waitedHalf so that just in case we somehow wind up here without having
// dumped the halfway stacks, we properly re-initialize the trace file.
final File stack = ActivityManagerService.dumpStackTraces(
!waitedHalf, pids, null, null, NATIVE_STACKS_OF_INTEREST);
// Give some extra time to make sure the stack traces get written.
// The system's been hanging for a minute, another second or two won't hurt much.
SystemClock.sleep(2000);
// Pull our own kernel thread stacks as well if we're configured for that
if (RECORD_KERNEL_THREADS) {
dumpKernelStackTraces();
}
// Trigger the kernel to dump all blocked threads, and backtraces on all CPUs to the kernel log
doSysRq('w');
doSysRq('l');
// Try to add the ERROR to the dropbox, but assuming that the ActivityManager
// itself may be deadlocked. (which has hAPPened, causing this statement to
// deadlock and the watchdog as a whole to be ineffective)
Thread dropboxThread = new Thread("watchdogWriteToDropbox") {
public void run() {
mActivity.addErrorToDropBox(
"watchdog", null, "system_server", null, null,
subject, null, stack, null);
}
};
dropboxThread.start();
try {
dropboxthread.join(2000); // wait up to 2 seconds for it to return.
} catch (InterruptedException ignored) {}
IActivitycontroller controller;
synchronized (this) {
controller = mController;
}
if (controller != null) {
Slog.i(TAG, "Reporting stuck state to activity controller");
try {
Binder.setDumpdisabled("Service dumps disabled due to hung system process.");
// 1 = keep waiting, -1 = kill system
int res = controller.systemNotResponding(subject);
if (res >= 0) {
Slog.i(TAG, "Activity controller requested to coninue to wait");
waitedHalf = false;
continue;
}
} catch (RemoteException e) {
}
}
// Only kill the process if the debugger is not attached.
if (Debug.isDebuggerConnected()) {
debuggerWasConnected = 2;
}
if (debuggerWasConnected >= 2) {
Slog.w(TAG, "Debugger connected: Watchdog is *not* killing the system process");
} else if (debuggerWasConnected > 0) {
Slog.w(TAG, "Debugger was connected: Watchdog is *not* killing the system process");
} else if (!allowRestart) {
Slog.w(TAG, "Restart not allowed: Watchdog is *not* killing the system process");
} else {
Slog.w(TAG, "*** WATCHDOG KILLING SYSTEM PROCESS: " + subject);
for (int i=0; i<blockedCheckers.size(); i++) {
Slog.w(TAG, blockedCheckers.get(i).getName() + " stack trace:");
StackTraceElement[] stackTrace
= blockedCheckers.get(i).getThread().getStackTrace();
for (StackTraceElement element: stackTrace) {
Slog.w(TAG, " at " + element);
}
}
Slog.w(TAG, "*** GOODBYE!");
Process.killProcess(Process.myPid());
System.exit(10);
}
waitedHalf = false;
}
}