Skip to content

any way to pipeline from generators or infinite pipelines? #17

@FarisHijazi

Description

@FarisHijazi

I'm using mpipe for computer vision processing from camera frames, it's exactly what I need, but the put() get() api is nonideal, and I can't figure out a way to have infinite tasks.

I read the documentation but I keep finding that you need to feed the tasks before running, I need to feed tasks while running

stage1.link(stage2)
...

pipe = Pipeline(...

for number in range(10):
    pipe.put(number)

I tried running a thread in the background to feed a multiprocessing.Queue but it just stops when the queue gets full, I concluded that you can't run put() while you're running get()

def fill_queue():
    while True:
        if not q.full():
            pipe.put(0)
        else:
            time.sleep(0.01)  # Rest for 10ms, we have a full queue
thread = Thread(target=fill_queue, args=())
thread.daemon = True


while True:
    if q.qsize():
       results... = q.get()

either way, something is very wrong

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions