Dataflow fails on Transform component in "TFX_Pipeline_for_Bert_Preprocessing"

[deep-diver]

@hanneshapke

I have tried out TFX_Pipeline_for_Bert_Preprocessing on GCP AI Platform Pipeline while Dataflow option turned on.

However, it failed, and I got some error messages.

tensorflow.python.framework.errors_impl.NotFoundError: Converting GraphDef to Graph has failed. The binary trying to import the GraphDef was built when GraphDef version was 440. The GraphDef was produced by a binary built when GraphDef version was 561. The difference between these versions is larger than TensorFlow's forward compatibility guarantee. The following error might be due to the binary trying to import the GraphDef being too old: Op type not registered 'CaseFoldUTF8' in binary running on beamapp-root-0218001007-4-02171610-z5bw-harness-3q6c. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed

any thoughts?

[hanneshapke]

@deep-diver

Hard to say what the issue is without seeing more details. Which component is failing? Can you maybe paste a longer error message, please?

[deep-diver]

@hanneshapke

The failure happens during TFX Transform component.

Here is the full message.
My guess is that it looks like BERT model provided from TF HUB is using non-compatible TF Graph version?

2021-02-18 09:13:43.360 JSTAn exception was raised when trying to execute the workitem 2017684186081758712 : Traceback (most recent call last): File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File "/usr/local/lib/python3.7/dist-packages/apache_beam/transforms/core.py", line 1616, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 782, in _infer_metadata_from_saved_model return _infer_metadata_from_saved_model_v1(saved_model_dir) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 793, in _infer_metadata_from_saved_model_v1 saved_model_dir, {})) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 426, in partially_apply_saved_transform_internal saved_model_dir, logical_input_map, tensor_replacement_map) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 271, in _partially_apply_saved_transform_impl input_map=input_map) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1462, in import_meta_graph **kwargs)[0] File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1486, in _import_meta_graph_with_return_elements **kwargs)) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/meta_graph.py", line 799, in import_scoped_meta_graph_with_return_elements return_elements=return_elements) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def producer_op_list=producer_op_list) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 497, in _import_graph_def_internal graph._c_graph, serialized, options) # pylint: disable=protected-access tensorflow.python.framework.errors_impl.NotFoundError: Converting GraphDef to Graph has failed. The binary trying to import the GraphDef was built when GraphDef version was 440. The GraphDef was produced by a binary built when GraphDef version was 561. The difference between these versions is larger than TensorFlow's forward compatibility guarantee. The following error might be due to the binary trying to import the GraphDef being too old: Op type not registered 'CaseFoldUTF8' in binary running on beamapp-root-0218001007-4-02171610-z5bw-harness-3q6c. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 649, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute op.start() File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start File "dataflow_worker/native_operations.py", line 54, in dataflow_worker.native_operations.NativeReadOperation.start File "apache_beam/runners/worker/operations.py", line 356, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 155, in apache_beam.runners.worker.operations.ConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 703, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/worker/operations.py", line 704, in apache_beam.runners.worker.operations.DoOperation.process File "apache_beam/runners/common.py", line 1215, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback raise exc.with_traceback(traceback) File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File "/usr/local/lib/python3.7/dist-packages/apache_beam/transforms/core.py", line 1616, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 782, in _infer_metadata_from_saved_model return _infer_metadata_from_saved_model_v1(saved_model_dir) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 793, in _infer_metadata_from_saved_model_v1 saved_model_dir, {})) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 426, in partially_apply_saved_transform_internal saved_model_dir, logical_input_map, tensor_replacement_map) File "/usr/local/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 271, in _partially_apply_saved_transform_impl input_map=input_map) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1462, in import_meta_graph **kwargs)[0] File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1486, in _import_meta_graph_with_return_elements **kwargs)) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/meta_graph.py", line 799, in import_scoped_meta_graph_with_return_elements return_elements=return_elements) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def producer_op_list=producer_op_list) File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 497, in _import_graph_def_internal graph._c_graph, serialized, options) # pylint: disable=protected-access RuntimeError: tensorflow.python.framework.errors_impl.NotFoundError: Converting GraphDef to Graph has failed. The binary trying to import the GraphDef was built when GraphDef version was 440. The GraphDef was produced by a binary built when GraphDef version was 561. The difference between these versions is larger than TensorFlow's forward compatibility guarantee. The following error might be due to the binary trying to import the GraphDef being too old: Op type not registered 'CaseFoldUTF8' in binary running on beamapp-root-0218001007-4-02171610-z5bw-harness-3q6c. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. [while running 'Analyze/ComputeDeferredMetadata[compat_v1=True]']

In case it is hard to read, here is the screenshot.