Pipeline keeps running on simple 'Stream Lookup' action.

[EricTonissen]

Anybody any idea why this Stream Lookup 'PATARTS' action doesn't continue?

AKKOORDJN output rows : 4990
QUERY PATARTS output rows : 12064

[bamaer]

not with only a partial screenshot of your pipeline

[EricTonissen]

Ok

[mattcasters]

Eric, consider simplifying the pipeline by splitting it up into smaller parts. (De-)Serialize to disk can help or create a database table with the results in-between. See also: https://hop.apache.org//manual/latest/best-practices/index.html#_size_matters

[dave-csc]

You most likely exceeded the buffer limit for a hop (10.000 rows by default), so your pipeline becomes deadlocked because it can't load all the necessary data for the lookup (i.e. the whole lookup query/table).

You may have better results by sorting the two streams and using a Merge join transform.

Hope this helps :)

[bamaer]

Yup. Reducing the number of row set size in your pipeline run configuration could also help.

[EricTonissen]

ByteArrayHashIndex.put() ends in an endless loop, if the array is full.

Better iterate the array only once. If it is full, then throw error.
// 
// If we are still here, that means that we are ready to put the value down...
// Where do we need to search for an empty spot in the index?
//
int len = index.length;
while (true) {
  indexPointer++;
  if (indexPointer >= len) {
    indexPointer = 0;
  }
  if (index[indexPointer] == null) {
    break;
  }
}

[EricTonissen]

Yes, because this one has the code for correct fieldname if they are duplicate

[hansva]

Hah, seems the previous code has the same potential issue... But because of the new "equalsByteArray" function it's just happening faster. Let me add some tests and harden that code... nice catch.

[hansva]

#6154

[EricTonissen]

And if I look in the StreamLookup.java, then the size of this array is 512 entries.

StreamLookup.java
if (data.hashIndex == null) {
data.hashIndex = new ByteArrayHashIndex(keyMeta);
}

Uses the contructor without size argument.

public ByteArrayHashIndex(IRowMeta keyRowMeta) {
    this(keyRowMeta, 512);
}

factor2size will end with 512.

int factor2Size = 1;
while (factor2Size < size) {
  factor2Size <<= 1; // Multiply by 2
}

[hansva]

Should be resolved with #6155. was able to create a test for the issue.

[sramazzina]

We are experiencing the same issue here @hansva. Lot of our working pipelines get stuck because of this. We will pull the latest changes, make some tests during the day and let you know

[sramazzina]

@hansva we tested the change here and everything worked fine. Thank you for your fixing it