Faster video processing

If you are using either a Raspberry Pi B+, 2, or 3 then threading is definitely the way to go.
Without threading the script can only really make use of one of the four cores.
This means in an ideal situation the processing could run up to four times as fast :)

We are also using cv2.VideoCapture(0) now with Formula Pi.
The Pi camera support has improved a lot since we wrote the DiddyBorg scripts and we have found it is faster than the older PiCamera based code.
This also has the advantage of the same code works with both the Pi Camera and a USB web camera :)
The OpenCV support for the Pi Camera can be setup by running this command:

sudo modprobe bcm2835-v4l2

The threading is a bit more complex as you have create the threads by hand.
This is not too difficult with Python, but understanding thread can be a bit tricky.

Below is the basic skeleton code we use for getting image processing code to run in multiple threads.
The actual processing code goes into the ProcessImage function inside the ImageProcessor class.

The ImageCapture class is responsible for getting frames from the camera.
It runs in its own thread, but uses very little processing time in practice.
What it does is wait until there is at least one processing thread waiting (held in processorPool).
It takes the oldest thread out of this pool and then gives it the next frame returned from calling cap.read and sends it an event notification.
This continues until the global running flag becomes False.
There should only be one of these threads running.

The ImageProcessor class gets passed the new frame and event notification each time we want it to process a frame from the ImageCapture thread.
Each runs in its own thread and adds itself back into the list of waiting threads (processorPool) after the ProcessImage function has finished.
We usually run four at the same time, but you should be able to have any number from one upwards :)

Our skeleton threading processing code:

# Imports
import cv2
import time
import threading

# Camera settings go here
imageWidth = ...
imageHeight = ...
frameRate = ...
processingThreads = 4

# Shared values
global running
global cap
global frameLock
global processorPool
running = True

# Setup the camera
cap = cv2.VideoCapture(0) 
cap.set(cv2.cv.CV_CAP_PROP_FRAME_WIDTH, imageWidth);
cap.set(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT, imageHeight);
cap.set(cv2.cv.CV_CAP_PROP_FPS, frameRate);
if not cap.isOpened():
    cap.open()

# Image processing thread, self-starting
class ImageProcessor(threading.Thread):
    def __init__(self, name, autoRun = True):
        super(ImageProcessor, self).__init__()
        self.event = threading.Event()
        self.eventWait = (2.0 * processingThreads) / frameRate
        self.name = str(name)
        print 'Processor thread %s started with idle time of %.2fs' % (self.name, self.eventWait)
        self.start()

    def run(self):
        # This method runs in a separate thread
        global running
        global frameLock
        global processorPool
        while running:
            # Wait for an image to be written to the stream
            self.event.wait(self.eventWait)
            if self.event.isSet():
                if not running:
                    break
                try:
                    self.ProcessImage(self.nextFrame)
                finally:
                    # Reset the event
                    self.nextFrame = None
                    self.event.clear()
                    # Return ourselves to the pool at the back
                    with frameLock:
                        processorPool.insert(0, self)
        print 'Processor thread %s terminated' % (self.name)

    def ProcessImage(self, image):
        # Processing for each image goes here
        ### TODO ###

# Image capture thread, self-starting
class ImageCapture(threading.Thread):
    def __init__(self):
        super(ImageCapture, self).__init__()
        self.start()

    # Stream delegation loop
    def run(self):
        # This method runs in a separate thread
        global running
        global cap
        global processorPool
        global frameLock
        while running:
            # Grab the oldest unused processor thread
            with frameLock:
                if processorPool:
                    processor = processorPool.pop()
                else:
                    processor = None
            if processor:
                # Grab the next frame and send it to the processor
                success, frame = cap.read()
                if success:
                    processor.nextFrame = frame
                    processor.event.set()
                else:
                    print 'Capture stream lost...'
                    running = False
            else:
                # When the pool is starved we wait a while to allow a processor to finish
                time.sleep(0.01)
        print 'Capture thread terminated'

# Create some threads for processing and frame grabbing
processorPool = [ImageProcessor(i+1) for i in range(processingThreads)]
allProcessors = processorPool[:]
captureThread = ImageCapture()

# Main loop, basically waits until you press CTRL+C
# The captureThread gets the frames and passes them to an unused processing thread
try:
    print 'Press CTRL+C to quit'
    while running:
        time.sleep(1)
except KeyboardInterrupt:
    print '\nUser shutdown'
except:
    e = sys.exc_info()
    print
    print e
    print '\nUnexpected error, shutting down!'

# Cleanup all processing threads
running = False
while allProcessors:
    # Get the next running thread
    with frameLock:
        processor = allProcessors.pop()
    # Send an event and wait until it finishes
    processor.event.set()
    processor.join()

# Cleanup the capture thread
captureThread.join()

# Cleanup the camera object
cap.release()

In laymens terms does the above code increase the frame rate in the Web UI? if so where would it go within the Diddyborg red web ui script?
if not it there a way to increase the frame rate and reduce the lag?

cheers
John