Internal Server Error in the user postinstall steps. #4
Description
First guess: my postinstall file might be too small...
pneyron@fgrenoble:~/rootfs$ kadeploy3 -a centos7-kameleon.env -k -m genepi-23.grenoble.grid5000.fr -d
Deployment #D-f40656a5-ccec-4d55-bcf8-d6c048b42eaf started
Grab the key file /home/pneyron/.ssh/authorized_keys
Grab the tarball file /home/pneyron/rootfs/centos7.tgz
Grab the postinstall file /home/pneyron/rootfs/env-postinstall.tgz
Launching a deployment on genepi-23.grenoble.grid5000.fr
Performing a Deploy[SetDeploymentEnvUntrusted] step
switch_pxe
reboot
* Performing a soft reboot on genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: ["ssh -q -F /etc/kadeploy3/keys/ssh_config root@genepi-23.grenoble.grid5000.fr \"if [ -f /bin/systemctl ]; then nohup /bin/systemctl reboot -f >/dev/null & else nohup /sbin/reboot -f >/dev/null & fi\""]
[dbg] STATUS: 255
* Performing a hard reboot on genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: ["/usr/local/bin/lanpower -c cycle -m genepi-23"]
[dbg] STATUS: 0
wait_reboot
send_key_in_deploy_env
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ cat - >>/root/.ssh/authorized_keys ] ; broadcast input file [ /var/cache/kadeploy/key_D_f40656a5_ccec_4d55_bcf8_d6c048b42eaf__home_pneyron__ssh_authorized_keys.data ]
[dbg] STATUS: 0
create_partition_table
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ tmp=`mktemp` && chmod 755 ${tmp} && cat - > $tmp && KADEPLOY_CLUSTER="genepi" KADEPLOY_DEPLOY_PART="/dev/sda3" KADEPLOY_BLOCK_DEVICE="/dev/sda" KADEPLOY_DEPLOY_PART_NUM="3" KADEPLOY_SWAP_PART_NUM="1" KADEPLOY_PROD_PART_NUM="2" KADEPLOY_TMP_PART_NUM="5" KADEPLOY_PREPOST_EXTRACTION_DIR="/rambin" KADEPLOY_TMP_DIR="/tmp" KADEPLOY_ENV="centos7-kameleon" KADEPLOY_ENV_KERNEL="/boot/vmlinuz-3.10.0-514.10.2.el7.x86_64" KADEPLOY_ENV_INITRD="/boot/initramfs-3.10.0-514.10.2.el7.x86_64.img" KADEPLOY_ENV_KERNEL_PARAMS="console=tty0 console=ttyS1,115200n8" KADEPLOY_ENV_HYPERVISOR="" KADEPLOY_ENV_HYPERVISOR_PARAMS="" KADEPLOY_OS_KIND="linux" KADEPLOY_PART_TYPE="83" KADEPLOY_FS_TYPE="ext4" KADEPLOY_ENV_EXTRACTION_DIR="/mnt/dest" ${tmp} ] ; broadcast input file [ /etc/kadeploy3/partitioning.sh ]
[dbg] STDOUT: File: `/dev/sda'
[dbg] STDOUT: Size: 0 Blocks: 0 IO Block: 4096 block special file
[dbg] STDOUT: Device: 11h/17d Inode: 7263 Links: 1 Device type: 8,0
[dbg] STDOUT: Access: (1660/brw-rw---T) Uid: ( 0/ root) Gid: ( 6/ disk)
[dbg] STDOUT: Access: 2017-03-09 16:05:23.599595927 +0000
[dbg] STDOUT: Modify: 2017-03-09 16:05:23.551595928 +0000
[dbg] STDOUT: Change: 2017-03-09 16:05:23.551595928 +0000
[dbg] STDOUT: Birth: -
[dbg] STATUS: 0
format_deploy_part
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ mkdir -p /mnt/dest; umount /dev/sda3 2>/dev/null; mkfs -t ext4 -b 4096 -O sparse_super,filetype,resize_inode,dir_index -q /dev/sda3 ]
[dbg] STATUS: 0
mount_deploy_part
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ mount /dev/sda3 /mnt/dest ]
[dbg] STATUS: 0
format_swap_part
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ mkswap /dev/sda1 ]
[dbg] STDOUT: Setting up swapspace version 1, size = 3905532 KiB
[dbg] STDOUT: no label, UUID=fabd92ad-2fe0-4de6-a108-338d1c8956a7
[dbg] STATUS: 0
End of step Deploy[SetDeploymentEnvUntrusted] after 102s
Performing a Deploy[BroadcastEnvKascade] step
send_environment
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: kascade -i /var/cache/kadeploy/tarball_D_f40656a5_ccec_4d55_bcf8_d6c048b42eaf__home_pneyron_rootfs_centos7_tgz.data -n /tmp/kascade-nodefile20170309-29202-10l8wxs -O 'tar --selinux --xattrs --xattrs-include='*' -C /mnt/dest -xz' -S 'ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A' --ok /tmp/kascade-okfile20170309-29202-17l6ad4 --ko /tmp/kascade-kofile20170309-29202-sffmpx -D taktuk -P '-d 0 -s'
[dbg] STDERR: syck has been removed, psych is used instead
[dbg] STDERR: W, [2017-03-09T17:05:49.708289 #5513] WARN -- : Connection failed on genepi-23.grenoble.grid5000.fr, retrying (1/10): Connection refused - connect(2) for "genepi-23.grenoble.grid5000.fr" port 29015
[dbg] STATUS: 0
* Broadcast time: 11s
manage_admin_post_install
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ tar --selinux --xattrs --xattrs-include='*' -C /rambin -xz ] ; broadcast input file [ /grid5000/preinstalls/preinstall-genepi.tgz ]
[dbg] STATUS: 0
manage_user_post_install
[dbg] -------------------------
[dbg] NODE: genepi-23.grenoble.grid5000.fr
[dbg] -------------------------
[dbg] COMMAND: taktuk --connector ssh -l root -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=publickey -o BatchMode=yes -A --self-propagate --dynamic 0 -o output="$type/$pid/$host/".length("$line").":$line"."\n" -o error="$type/$pid/$host/".length("$line").":$line"."\n" -o status="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o connector="$type/$pid/$host/".length("$command").":$command".length("$line").":$line"."\n" -o state="$type/$pid/$host/".length("$command").":$command".length("$line").":$line".length("$peer").":$peer"."\n" -o info -o message -o taktuk -m genepi-23.grenoble.grid5000.fr broadcast exec [ tar --selinux --xattrs --xattrs-include='*' -C /rambin -xz ] ; broadcast input file [ /var/cache/kadeploy/postinstall_D_f40656a5_ccec_4d55_bcf8_d6c048b42eaf__home_pneyron_rootfs_env_postinstall_tgz.data ]
[dbg] STATUS: 0
---
Deployment status (251s):
[BroadcastEnvKascade-manage_user_post_install] ~135s (OK)
genepi-23.grenoble.grid5000.fr
---
[Internal Server Error]
---- NoMethodError ----
undefined method `join' for nil:NilClass
---- Stack trace ----
/usr/lib/ruby/vendor_ruby/kadeploy3/server/parallel_ops.rb:80:in `taktuk_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:293:in `block (2 levels) in parallel_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:260:in `parallel_op'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:286:in `block in parallel_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:301:in `call'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:301:in `parallel_exec'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:2203:in `block in ms_manage_user_post_install'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:2192:in `each'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:2192:in `ms_manage_user_post_install'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/microsteps.rb:64:in `run'
/usr/lib/ruby/vendor_ruby/kadeploy3/server/automata.rb:460:in `block (2 levels) in run_task'
---------------------