How to use the pre-trained model

[seviezhou]

Can you provide some instructions on how to use the pre-trained model to disassemble a new binary file?

[yixiaoxian]

I have the same question. I attempted to use fairseq-interactive to emulate the method, which, however, requires to specify the language pair. So, what is the language type of binaries?

[peikexin9]

Guys, pardon me for a couple of days. I'll set up the scripts asap. Thanks for the patience!

[peikexin9]

@seviezhou @yixiaoxian Thanks for your patience!! I have updated some scripts for you to play with the fine-tuned model. See here. Happy new year :-)

[seviezhou]

I tried your script, but there are two problems.

First, the path of dict.txt seems to be wrong, and I fix it by changing this line:

roberta = RobertaModel.from_pretrained('checkpoints/finetune_msvs_funcbound_64', 'checkpoint_best.pt',
                                       'data-bin/funcbound_msvs_64/data', bpe=None, user_dir='finetune_tasks')

Second, when I run python scripts/play/play_func_bound.py, there is an error:

Ground Truth:
33 c9 e9 81 61 0 0 cc cc cc cc cc cc cc cc cc 8b d ea 1c 23 0 33 c0 8b d0 85 c9 74 23 4c 8d 5 db 1c 23 0 66 66 66 f 1f 84 0 0 0 0 0 3b c8 f 4f c1 41 8b 4c 90 4 48 ff c2 85 c9 75 ef f3 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 8b 5 aa 1c 23 0 33 d2 85 c0 74 17 4c 8d 5 9d 1c 23 0 3b c1 74 f 41 8b 44 90 4 48 ff c2 85 c0 75 f0 33 c0 c3 b8 1 0 0 0 c3 cc cc cc cc 48 83 ec 28 48 8b 1 80 38 0 75 f 33 d2 8d 4a a e8 8a fd 12 0 e8 25 2 0 0 48 83 c4 28 c3 40 53 56 57 41 55 48 83 ec 28 4c 8b 5 f 19 23 0 33 f6 4c 8d d 6 19 23 0 44 8b ee 8b de 8b d6 4d 85 c0 74 2d 49 8b f8 f 1f 80 0 0 0 0 48 83 c9 ff 33 c0 f2 ae 49 8b 7c d1 8 48 f7 d1 48 ff c9 3b cb f 4f d9 48 ff c2 41 ff c5 48 85 ff 75 dd 8b 5 8b ed 24 0 ff c3 3b c3 7d 46 4d 85 c0 f 84 65 1 0 0 49 8b c1 48 8d 3d ae 18 23 0 48 8b 8 e8 e6 d 0 0 83 3d f 7a 27 0 0 7e a 33 d2 8d 4a a e8 f3 fc 12 0 48 ff c6 48 83 3c f7 0 48 8d 4 f7 75 d7 48 83 c4 28 41 5d 5f 5e 5b c3 8b d 89 59 27 0 ff c0 48 89 6c 24 58 99 4c 89 7c 24 20 8b fe 8b ee f7 fb 44 8b f8 41 8b c5 99 41 f7 ff 85 d2 40 f 95 c7 3 f8 89 7c 24 50 85 c9 f 85 e7 0 0 0 4c 89 74 24 68 44 8b f7 4c 89 64 24 60 45 f af f7 f 1f 0 41 3b ee f 8d c0 0 0 0 8b c5 99 41 f7 ff 8b ca f af cf 3 c8 41 3b cd f 8d ee 0 0 0 8d 45 1 44 8b e6 99 41 f7 ff 48 63 c1 49 8b 3c c1 48 8b cf 85 d2 41 f 94 c4 83 ca ff e8 5f d4 12 0 39 35 79 58 27 0 75 4e 39 35 f1 79 27 0 74 1a 8b 5 75 56 27 0 39 35 1b 56 27 0 b9 1 0 0 0 f 44 c1 89 5 61 56 27 0 b8 1 0 0 0

F - - - - - - R - - - - - - - - F - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - R - - - - - - - - - - - - F - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - R - - - F - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - R - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IDA-PRO:
33 c9 e9 81 61 0 0 cc cc cc cc cc cc cc cc cc 8b d ea 1c 23 0 33 c0 8b d0 85 c9 74 23 4c 8d 5 db 1c 23 0 66 66 66 f 1f 84 0 0 0 0 0 3b c8 f 4f c1 41 8b 4c 90 4 48 ff c2 85 c9 75 ef f3 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 8b 5 aa 1c 23 0 33 d2 85 c0 74 17 4c 8d 5 9d 1c 23 0 3b c1 74 f 41 8b 44 90 4 48 ff c2 85 c0 75 f0 33 c0 c3 b8 1 0 0 0 c3 cc cc cc cc 48 83 ec 28 48 8b 1 80 38 0 75 f 33 d2 8d 4a a e8 8a fd 12 0 e8 25 2 0 0 48 83 c4 28 c3 40 53 56 57 41 55 48 83 ec 28 4c 8b 5 f 19 23 0 33 f6 4c 8d d 6 19 23 0 44 8b ee 8b de 8b d6 4d 85 c0 74 2d 49 8b f8 f 1f 80 0 0 0 0 48 83 c9 ff 33 c0 f2 ae 49 8b 7c d1 8 48 f7 d1 48 ff c9 3b cb f 4f d9 48 ff c2 41 ff c5 48 85 ff 75 dd 8b 5 8b ed 24 0 ff c3 3b c3 7d 46 4d 85 c0 f 84 65 1 0 0 49 8b c1 48 8d 3d ae 18 23 0 48 8b 8 e8 e6 d 0 0 83 3d f 7a 27 0 0 7e a 33 d2 8d 4a a e8 f3 fc 12 0 48 ff c6 48 83 3c f7 0 48 8d 4 f7 75 d7 48 83 c4 28 41 5d 5f 5e 5b c3 8b d 89 59 27 0 ff c0 48 89 6c 24 58 99 4c 89 7c 24 20 8b fe 8b ee f7 fb 44 8b f8 41 8b c5 99 41 f7 ff 85 d2 40 f 95 c7 3 f8 89 7c 24 50 85 c9 f 85 e7 0 0 0 4c 89 74 24 68 44 8b f7 4c 89 64 24 60 45 f af f7 f 1f 0 41 3b ee f 8d c0 0 0 0 8b c5 99 41 f7 ff 8b ca f af cf 3 c8 41 3b cd f 8d ee 0 0 0 8d 45 1 44 8b e6 99 41 f7 ff 48 63 c1 49 8b 3c c1 48 8b cf 85 d2 41 f 94 c4 83 ca ff e8 5f d4 12 0 39 35 79 58 27 0 75 4e 39 35 f1 79 27 0 74 1a 8b 5 75 56 27 0 39 35 1b 56 27 0 b9 1 0 0 0 f 44 c1 89 5 61 56 27 0 b8 1 0 0 0

Traceback (most recent call last):
  File "scripts/play/play_func_bound.py", line 123, in <module>
    tokens, func_start, func_end = predict_color('msvs_64_O2_vim', roberta, start_idx=0, end_idx=510)
  File "scripts/play/play_func_bound.py", line 97, in predict_color
    logprobs = model.predict('funcbound', encoded_tokens)
  File "/venv/XDA/fairseq/models/roberta/hub_interface.py", line 96, in predict
    logits = self.model.classification_heads[head](features)
  File "/venv/lib/python3.6/site-packages/torch/nn/modules/container.py", line 286, in __getitem__
    return self._modules[key]
KeyError: 'funcbound'

Can you please look at it? Thank you!

[peikexin9]

Hi Zhou, It runs with no problem on my end.
Did you make sure you download the finetuned weights (described here) instead of the pretrained weights (described here) to checkpoints/finetune_msvs_funcbound_64?
You should still put data-bin/funcbound_msvs_64 (not data-bin/funcbound_msvs_64/data) in that line as the label dictionary ('data-bin/funcbound_msvs_64/label)' should also be there for the model to predict.
Given your second problem, I highly doubt you are using the pretrained model weights, which are not for function boundary prediction but masked byte prediction (see the paper for details).

Let me know if this could fix your problem :-)

[seviezhou]

Thank you, it works now.

[yixiaoxian]

Thanks for the script, but all the test samples seem to be compiled with msvc, so I wonder if the finetuned parameters you provide can only be applied to msvc-binaries? Can I also use those parameters to analyze binaries compiled with gcc?

[peikexin9]

Hi @yixiaoxian, I think you can also try to apply it to GCC-compiled binaries and see how it works?
I can also upload the finetuned model for GCC for you. Sorry, I am a little busy recently but I will update as soon as I can.
If you are interested, you can also finetune the pretrained model for GCC binaries you collected -- I have included the scripts.
Thanks for the interest anyway! Happy new year :-)

[jupiterepoch]

Hi, I'm trying to use your tool on my own binaries (both gcc & msvc). I was wondering if you can release your scripts for labelling the ground truth? i.e. matching function starts and ends to exact binary bytes. Thank you!

[peikexin9]

Hi @jupiterepoch, thanks for your interest! I just uploaded a sample script at scripts/play/eval_pair_bound.py. See if that helps?

[jupiterepoch]

@peikexin9 Hi, you mentioned above about your pretrained model for GCC (the checkpoints), I wonder if it is possible for you to release it (and also clang compiled versions if you have those)? I would be very grateful, as pretraining on my side seems to take forever.