Python subprocess pipe block

sqd Source

For some reasons, in this example consumer seems to be blocked (because it's waiting for EOF?). But the writing end of the pipe should have been closed.

import os, subprocess, threading
(r, w) = os.pipe()

def producer():
    print '--producer started'
    subprocess.Popen(['echo', '123'], stdout=w).wait()
    print '--producer finished'
    os.close(w)
    print '--write pipe closed'

def consumer():
    print '--receiver started'
    subprocess.Popen(['cat'], stdin=r).wait()
    print '--receiver finished'
    os.close(r)
    print '--read pipe closed'

threading.Thread(target=producer).start()
threading.Thread(target=consumer).start()
pythonlinuxmultithreadingpipe

Answers

answered 2 months ago mata #1

The problem here is that all processes inherit all open file descriptors, and that there will only be an EOF on the read end of the pipe when the write end has been closed in all processes. Because the cat process itself keeps the write end open calling os.close(w) in your main process is not enough.

For this case Popen() has the close_fds argument, which when set to True will cause all open file descriptors except those used for stdin, stderr and stout to be closed immediately after creating the subprocess. If you do that the your example should work as expected.

On python2 close_fds=False is the default, in python3 this was changed to close_fds=True, and additionally since python3.4 file desciptors that should be inherited need to be marked as inheritable explicitly using os.set_inheritable(). That reduces this kind of problem.

answered 2 months ago Jonah #2

The reason your program never exits is that you're waiting on cat, except cat is waiting for an EOF that never comes.

when you call subprocess.Popen(['cat'], stdin=r).wait(), it reads from the file descriptor and prints it to stdout because you didn't redirect stdout. This is why you still see '123' on the screen.

Consider either of these options instead: Use a queue:

import threading
from Queue import Queue
q= Queue()

def producer():
    print '--producer started'
    q.put('123')
    print '--producer finished'

def consumer():
    print '--receiver started'
    print q.get()
    print '--receiver finished'

threading.Thread(target=producer).start()
threading.Thread(target=consumer).start()

or use os.pipe but use read and write instead

import os, subprocess, threading
(r, w) = os.pipe()

def producer():
    print '--producer started'
    os.write(w, '123')
    print '--producer finished'
    os.close(w)
    print '--write pipe closed'

def consumer():
    print '--receiver started'
    print os.read(r, 3)
    print '--receiver finished'
    os.close(r)
    print '--read pipe closed'

threading.Thread(target=producer).start()
threading.Thread(target=consumer).start()

comments powered by Disqus