Kelly's Space

Coding 'n stuff

Monthly Archives: May 2008

Random process hangs while reading output

I’ve been debugging an intermittent problem that occurs under load where launched processes exit and threads reading the output from the process hang while reading the output.  Below is a common snippet of code to launch a process, and redirect the output to separate threads (to avoid deadlock that occurs if the client process fills the output buffer and cannot exit because the host process isn’t reading the pipe):

        public void LaunchProgram(string programName)
        {
            Process p = new Process();
            ProcessStartInfo psi = new ProcessStartInfo();
            psi.FileName = programName;

            psi.CreateNoWindow = true;
            psi.WindowStyle = ProcessWindowStyle.Hidden;
            psi.RedirectStandardError = true;
            psi.RedirectStandardOutput = true;
            psi.UseShellExecute = false;
            psi.RedirectStandardInput = false;
            psi.WorkingDirectory = Directory.GetCurrentDirectory();
            p.StartInfo = psi;
            p.Start();

            OutputReader stdOutReader = new OutputReader(p.StandardOutput);
            Thread stdOutThread = new Thread(new ThreadStart(stdOutReader.ReadOutput));
            stdOutThread.Start();
            OutputReader stdErrReader = new OutputReader(p.StandardError);
            Thread stdErrThread = new Thread(new ThreadStart(stdErrReader.ReadOutput));
            stdErrThread.Start();

            p.WaitForExit();

            stdOutThread.Join();
            stdErrThread.Join();
        }

 

The OutputReader class looks like this:

    public class OutputReader
    {
        StreamReader stream;

        public OutputReader(StreamReader stream)
        {
            this.stream = stream;
        }

        public void ReadOutput()
        {
            string line = null;

            while ((line = stream.ReadLine()) != null)
            {
                // do something with the output
            }
        }
    }

The problem occurs when the LaunchProgram method is called in a multithreaded app. 

The core issue is that the Process class creates pipes for redirecting the output using CreatePipe and then calls CreateProcess with the inheritHandles parameter set to true.  Under load there is a race condition where a process can inherit handles that it shouldn’t.  In the case I was debugging, each of the programs were synchronizing with a database for shared resources that were only released when the process exited.  Since another process had a copy of the pipes the OutputReader thread was still waiting for data to come in because there was still an open pipe handle on the other side.  The other executable was deadlocked because it was waiting for shared resources held by the first process.

The fix for this is to make sure that the CreatePipe and CreateProcess calls happen synchronously in a process.  An easy fix is to add a static lock object, and lock it before the call to p.Start().  That is:

static object ProcessLockObj = new object();

public void LaunchProgram(string programName)
        {
            Process p = new Process();
            ProcessStartInfo psi = new ProcessStartInfo();
            psi.FileName = programName;

            psi.CreateNoWindow = true;
            psi.WindowStyle = ProcessWindowStyle.Hidden;
            psi.RedirectStandardError = true;
            psi.RedirectStandardOutput = true;
            psi.UseShellExecute = false;
            psi.RedirectStandardInput = false;
            psi.WorkingDirectory = Directory.GetCurrentDirectory();
            p.StartInfo = psi;

            lock (ProcessLockObj)
            {
                p.Start();
            }

            OutputReader stdOutReader = new OutputReader(p.StandardOutput);
            Thread stdOutThread = new Thread(new ThreadStart(stdOutReader.ReadOutput));
            stdOutThread.Start();
            OutputReader stdErrReader = new OutputReader(p.StandardError);
            Thread stdErrThread = new Thread(new ThreadStart(stdErrReader.ReadOutput));
            stdErrThread.Start();

            p.WaitForExit();

            stdOutThread.Join();
            stdErrThread.Join();
        }

The problem can also manifest itself if you are not using the Process class, but are calling CreateProcess directly.  If you are doing this, then make sure that you place CreateProcess and the CreatePipe calls in a critical section.

Lots of debugging and searching narrowed down the problem, and then the following article explained what was going on: http://support.microsoft.com/kb/315939

Advertisements