AllenNLP issues

[pseudomonas]

Microbert's allennlp version seems to suffer from this issue allenai/allennlp#5666 which per the comments on that issue, is fixed by running

pip install cached-path==1.1.2

immediately after running pip install -r requirements.txt in the installation instructions.

But the need to use that allennlp version I think might also be getting me bitten by allenai/allennlp#4317 or similar; while trying to pretrain a Bert on Coptic with python main.py bert train , I'm getting

2023-02-10 20:54:36,408 - INFO - allennlp.common.params - dataset_reader.readers.mlm.token_indexers.tokens.token_min_padding_length = 0
2023-02-10 20:54:36,408 - INFO - allennlp.common.params - dataset_reader.readers.mlm.token_indexers.tokens.model_name = berts/coptic/mlm-xpos-parser_layers-3_heads-5_hidden-100
2023-02-10 20:54:36,408 - INFO - allennlp.common.params - dataset_reader.readers.mlm.token_indexers.tokens.namespace = tags
2023-02-10 20:54:36,408 - INFO - allennlp.common.params - dataset_reader.readers.mlm.token_indexers.tokens.max_length = None
2023-02-10 20:54:36,408 - INFO - allennlp.common.params - dataset_reader.readers.mlm.token_indexers.tokens.tokenizer_kwargs = None
2023-02-10 20:54:36,413 - CRITICAL - root - Uncaught exception
Traceback (most recent call last):
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/data/tokenizers/pretrained_transformer_tokenizer.py", line 83, in __init__
    self._reverse_engineer_special_tokens("a", "b", model_name, tokenizer_kwargs)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/data/tokenizers/pretrained_transformer_tokenizer.py", line 136, in _reverse_engineer_special_tokens
    assert dummy_a != dummy_b
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pycharm_project_494/main.py", line 131, in <module>
    top()
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/pycharm_project_494/embur/commands/bert.py", line 147, in train
    model = train_model_from_file(config.pretrain_jsonnet, config.experiment_dir, overrides=overrides)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/commands/train.py", line 177, in train_model_from_file
    return train_model(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/commands/train.py", line 258, in train_model
    model = _train_worker(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/commands/train.py", line 494, in _train_worker
    train_loop = TrainModel.from_params(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 604, in from_params
    return retyped_subclass.from_params(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 636, in from_params
    kwargs = create_kwargs(constructor_to_inspect, cls, params, **extras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 206, in create_kwargs
    constructed_arg = pop_and_construct_arg(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 314, in pop_and_construct_arg
    return construct_arg(class_name, name, popped_params, annotation, default, **extras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 348, in construct_arg
    result = annotation.from_params(params=popped_params, **subextras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 604, in from_params
    return retyped_subclass.from_params(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 636, in from_params
    kwargs = create_kwargs(constructor_to_inspect, cls, params, **extras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 206, in create_kwargs
    constructed_arg = pop_and_construct_arg(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 314, in pop_and_construct_arg
    return construct_arg(class_name, name, popped_params, annotation, default, **extras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 394, in construct_arg
    value_dict[key] = construct_arg(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 348, in construct_arg
    result = annotation.from_params(params=popped_params, **subextras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 604, in from_params
    return retyped_subclass.from_params(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 636, in from_params
    kwargs = create_kwargs(constructor_to_inspect, cls, params, **extras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 206, in create_kwargs
    constructed_arg = pop_and_construct_arg(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 314, in pop_and_construct_arg
    return construct_arg(class_name, name, popped_params, annotation, default, **extras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 394, in construct_arg
    value_dict[key] = construct_arg(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 348, in construct_arg
    result = annotation.from_params(params=popped_params, **subextras)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 604, in from_params
    return retyped_subclass.from_params(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/common/from_params.py", line 638, in from_params
    return constructor_to_call(**kwargs)  # type: ignore
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/data/token_indexers/pretrained_transformer_mismatched_indexer.py", line 58, in __init__
    self._matched_indexer = PretrainedTransformerIndexer(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/data/token_indexers/pretrained_transformer_indexer.py", line 56, in __init__
    self._allennlp_tokenizer = PretrainedTransformerTokenizer(
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/data/tokenizers/pretrained_transformer_tokenizer.py", line 87, in __init__
    self._reverse_engineer_special_tokens("1", "2", model_name, tokenizer_kwargs)
  File "/storage/condaenvs/thisenv/lib/python3.9/site-packages/allennlp/data/tokenizers/pretrained_transformer_tokenizer.py", line 136, in _reverse_engineer_special_tokens
    assert dummy_a != dummy_b
AssertionError

Process finished with exit code 1

[lgessler]

Argh, yes, I remember this issue coming up. So, as I recall, that issue can be solved by setting the PretrainedTransformerTokenizer's verification_tokens parameter to something like ["ⲙ", "ⲛ"]. Could you check whether that resolves the issue?

[pseudomonas]

Doesn't seem to help, in [conda env]/lib/python3.9/site-packages/allennlp/data/tokenizers/pretrained_transformer_tokenizer.py …

    def __init__(
        self,
        model_name: str,
        add_special_tokens: bool = True,
        max_length: Optional[int] = None,
        tokenizer_kwargs: Optional[Dict[str, Any]] = None,
        verification_tokens: Optional[Tuple[str, str]] = ["ⲙ", "ⲛ"],
    ) -> None:

(and the same if I try the tuple ("ⲙ", "ⲛ"). and the same if I try with a variety of different single coptic characters)

[pseudomonas]

I'm wondering if this is a problem because I'm trying to train a new model rather than using an existing one which already has a tokenizer?

(I've been running main.py -l coptic --debug bert train but it fails in the same way without the debug flag)

[lgessler]

Can you verify that the tokenizer is actually getting data? I think if config is subtly off and zero training data is read, then the program can proceed up to this point. To check this, you can check the tokenizer's serialized files (should be at berts/<language>/vocab.txt to ensure that the tokenizer actually learned something sensible.

[pseudomonas]

Apparently not. Only seems to know the preconfigured tokens.

microbert/berts/coptic/mlm-xpos-parser_layers-3_heads-5_hidden-100 $ cat vocab.txt 
[PAD]
[CLS]
[SEP]
[UNK]
[MASK]

[pseudomonas]

Data is definitely present in what looks like the right place.

microbert/data/coptic/corpora/life-cyrus/life.cyrus_CONLLU $ head life.cyrus.01.conllu 
# newdoc id = life.cyrus:life.cyrus.01
# sent_id = life_cyrus-life_cyrus_01_s0001
# text = ⲡⲃⲓⲟⲥ ⲁⲩⲱ ⲧⲡⲟⲗⲓⲧⲉⲓⲁ ⲙⲡⲉⲛⲡⲉⲧⲟⲩⲁⲁⲃ ⲛⲉⲓⲱⲧ ⲉⲧⲧⲁⲓⲏⲩ ⲕⲁⲧⲁⲥⲙⲟⲧ ⲛⲓⲙ ⲡϩⲁⲅⲓⲟⲥ ⲁⲡⲁ ⲕⲩⲣⲟⲥ · ⲡⲧⲉⲗⲓⲟⲥ ⲛⲁⲙⲉ ⲛⲧⲁϥϩⲓⲥⲧⲟⲣⲓⲍⲉ ⲙⲙⲟϥ · ⲛϭⲓⲁⲡⲁ ⲡⲁⲙⲃⲱ ⲡⲉⲡⲣⲉⲥⲃⲩⲧⲉⲣⲟⲥ ⲛⲧⲉⲕⲕⲗⲏⲥⲓⲁ ⲛϣⲓⲏⲧ ⲛⲧⲁϥⲙⲧⲟⲛ ⲙⲙⲟϥ ⲛϭⲓⲡⲉⲧⲟⲩⲁⲁⲃ · ⲁⲡⲁ ⲕⲩⲣⲟⲥ ϩⲛⲥⲟⲩ
ϣⲙⲟⲩⲛ ⲙⲡⲉⲃⲟⲧ ⲉⲡⲏⲡ ϩⲛⲟⲩⲉⲓⲣⲏⲛⲏ ⲛⲧⲉⲡⲛⲟⲩⲧⲉ ·
# text_en = ...
1-2     ⲡⲃⲓⲟⲥ   _       _       _       _       _       _       _       _
1       ⲡ       ⲡ       DET     ART     Definite=Def|Gender=Masc|Number=Sing|PronType=Art       2       det     _       Entity=(abstract|Orig=Ⲡ
2       ⲃⲓⲟⲥ    ⲃⲓⲟⲥ    NOUN    N       _       0       root    _       Orig=ⲂⲒⲞⲤ
3       ⲁⲩⲱ     ⲁⲩⲱ     CCONJ   CONJ    _       5       cc      _       Orig=ⲀⲨⲰ
4-5     ⲧⲡⲟⲗⲓⲧⲉⲓⲁ       _       _       _       _       _       _       _       _
4       ⲧ       ⲡ       DET     ART     Definite=Def|Gender=Fem|Number=Sing|PronType=Art        5       det     _       Entity=(abstract|Orig=Ⲧ

[lgessler]

Oh, have you run python main.py -l coptic data prepare-mlm?

[pseudomonas]

No; was I supposed to?

[lgessler]

Yeah, that command takes the native format for MLM data (varies by language) and processes it into the unified format used by the data reading code. Please try running that and see if that resolves the issue--sorry that wasn't made clear in the readme.

[pseudomonas]

microbert $ python main.py -l coptic data prepare-mlm
Traceback (most recent call last):
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 857, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/onnx/config.py", line 35, in <module>
    from PIL import Image
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/PIL/Image.py", line 103, in <module>
    from . import _imaging as core
ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by [path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/PIL/../../.././libLerc.so)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "[path]/microbert/main.py", line 7, in <module>
    import_module_and_submodules("allennlp_models")
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp/common/util.py", line 376, in import_module_and_submodules
    import_module_and_submodules(subpackage, exclude=exclude)
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp/common/util.py", line 359, in import_module_and_submodules
    module = importlib.import_module(package_name)
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp_models/classification/__init__.py", line 2, in <module>
    from allennlp_models.classification.models import *
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp_models/classification/models/__init__.py", line 1, in <module>
    from allennlp_models.classification.models.biattentive_classification_network import (
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp_models/classification/models/biattentive_classification_network.py", line 10, in <module>
    from allennlp.data import TextFieldTensors, Vocabulary
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp/data/__init__.py", line 1, in <module>
    from allennlp.data.data_loaders import (
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp/data/data_loaders/__init__.py", line 2, in <module>
    from allennlp.data.data_loaders.multiprocess_data_loader import MultiProcessDataLoader, WorkerError
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp/data/data_loaders/multiprocess_data_loader.py", line 19, in <module>
    from allennlp.data.data_loaders.data_collator import DataCollator, DefaultDataCollator
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/allennlp/data/data_loaders/data_collator.py", line 3, in <module>
    from transformers.data.data_collator import DataCollatorForLanguageModeling
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/data/__init__.py", line 19, in <module>
    from .data_collator import (
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/data/data_collator.py", line 20, in <module>
    from ..models.bert import BertTokenizer, BertTokenizerFast
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/models/__init__.py", line 19, in <module>
    from . import (
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/models/layoutlm/__init__.py", line 22, in <module>
    from .configuration_layoutlm import LAYOUTLM_PRETRAINED_CONFIG_ARCHIVE_MAP, LayoutLMConfig
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/models/layoutlm/configuration_layoutlm.py", line 22, in <module>
    from ...onnx import OnnxConfig, PatchingSpec
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 847, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "[path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 859, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.onnx.config because of the following error (look up to see its traceback):
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by [path]/miniconda3/envs/trainenv/lib/python3.9/site-packages/PIL/../../.././libLerc.so)

[pseudomonas]

OK, that was fixed with appending […]/miniconda3/envs/trainenv/lib to LD_LIBRARY_PATH

…and with that, it all seems to run correctly at least for the --debug mode.

[lgessler]

Great, sorry about the hiccups, glad you got it working correctly in the end! Let me know if you have any other questions.