k8s is not accessible

[putcn]

after manually start dnsmasq in container, now cluster workers are able to boot and install. but after worker rebooted, I'm not able to config kubectl to connect to k8s api server.

First issue is: looks like the vars defined there are not properly substituted in setup_kubectl.bash with my cluster-desc.yaml.
like KUBE_MASTER_HOSTNAME, it was 00-25-90-c0-f7-80, but it supposed to be 08-00-27-00-B2-98 or kube-master as in my yaml file. And also the BS_IP, which is wrong. I tracked common.sh, looks it should have worked properly, but still, the output is wrong, not sure why.

Then I manually updated these vars, it still does not work.
kube-master names are not resolved by dnsmasq. "kube-master" or "08-00-27-00-B2-98" is supposed to be resolved by dnsmasq to IP address, but I don't see these names are defined anywhere, and I checked dnsmasq.conf, it does have a setting for addn-hosts=/bsroot/config/dnsmasq.hosts, but there isn't such a file in config folder. how is name service defined if this host file is not generated?

my cluster-desc.yaml is as follows

bootstrapper: 10.0.3.3
subnet: 10.0.3.0
netmask: 255.255.255.0
iplow: 10.0.3.7
iphigh: 10.0.3.252
routers: [10.0.3.1]
broadcast: 10.0.3.255
nameservers: [10.0.3.3]
upstreamnameservers: [8.8.8.8, 8.8.4.4]
domainname: "ail.unisound.com"
dockerdomain: "bootstrapper"
k8s_service_cluster_ip_range: 10.100.0.0/24
k8s_cluster_dns: 10.100.0.10

#start pxe?
start_pxe: y

# Flannel backend only support "host-gw", "vxlan" and "udp" for now.
flannel_backend: "host-gw"

# coreos_channel can be configured as stable, alpha, beta.
coreos_channel: "stable"

# coreos_version can be configured as the special version num or "current"
coreos_version: "1122.2.0"

# centos version
download_kernel: y
# configure mirror_site, sunch as:
#   http://mirrors.163.com/centos
mirror_site: http://10.0.3.1/centos
# Configure the centos version for isos and rpms
centos_version: "7.4.1708"

# gpu drivers version
set_gpu: n
gpu_drivers_version: "375.26"

ingress_hostnetwork: true

# Ntpserver set_ntp option for the cluster configuration.
set_ntp: y
# Set DNSMASQ DHCP least time
lease: "infinite"

#OS type: CentOS or CoreOS
os_name: "CentOS"
# set_yum_repo: "mirrors.163.com"
set_yum_repo: "mirrors.aliyun.com"

# kube master ip, there should be cluster ip
kube_master_ip:
    - "10.100.0.1"
    - "10.0.3.4"
    - "10.0.3.3"
kube_master_dns:
    - "aa-bb-cc-dd"

coreos:
  reboot_strategy: "etcd-lock"
  start_time: "03:00"
  time_length: "3h"

ceph:
  zap_and_start_osd: n
  osd_journal_size: 5000

images:
  hyperkube: "gcr.io/google_containers/hyperkube-amd64:v1.6.2"
  pause: "gcr.io/google_containers/pause-amd64:3.0"
  flannel: "quay.io/coreos/flannel:v0.7.1-amd64"
  ingress: "gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3"
  default_backend: "gcr.io/google_containers/defaultbackend:1.0"
  kubedns: "gcr.io/google_containers/kubedns-amd64:1.9"
  kubednsmasq: "gcr.io/google_containers/kube-dnsmasq-amd64:1.4"
  healthz: "gcr.io/google_containers/exechealthz-amd64:1.2"
  addon_manager: "gcr.io/google_containers/kube-addon-manager-amd64:v6.4-beta.1"
  ceph: "typhoon1986/ceph-daemon:tag-build-master-jewel-ubuntu-14.04-fix370"
  ntp: "redaphid/docker-ntp-server"
  heapster: "kubernetes/heapster:canary"
  grafana: "lupan/heapster_grafana:v2.6.0-2"
  influxdb: "lupan/heapster_influxdb:v0.5"
  dashboard: "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0"


# if you wan't use mac as hostname, please delete cur_host_name or 
# set them to ""
nodes:
  - mac: "08:00:27:00:B2:98"
    cur_host_name: "kube-master"
    ceph_monitor: n
    kube_master: y
    etcd_member: y
    ingress_label: n
  - mac: "0c:c4:7a:82:c5:bc"
    cur_host_name: "etcd0"
    ceph_monitor: n
    kube_master: n
    etcd_member: y
    ingress_label: n
  - mac: "0c:c4:7a:82:c5:b8"
    cur_host_name: "etcd1"
    ceph_monitor: n
    kube_master: n
    etcd_member: y
    ingress_label: y
  - mac: "00:25:90:c0:f6:ee"
    cur_host_name: "node0"
    ceph_monitor: n
    kube_master: n
    etcd_member: n
    ingress_label: n
  - mac: "00:25:90:c0:f6:d6"
    cur_host_name: "node1"
    ceph_monitor: n
    kube_master: n
    etcd_member: n
    ingress_label: n

[putcn]

i have an idea regarding this issue. Since we are trying not to assign static ip addresses for those machines who have special roles( kube master, etcd memeberes, etc.), we can dynamically add hostname and ip address to dnsmasq's hosts file by utilizing dnsmasq's dhcp_script and hostsdir.

Whenever a machine joins the network and asks for an ip address. dhcp_script will be called with the ip assigned to it, the machine's mac address, etc. we can maintain the hosts mapping file with this info and cluster-desc.yaml's predefined node list (either with host name converted from mac, or with predefined cur_host_name). dnsmasq will refresh its nameservice when the hosts file changed.

after this, whenever another machine is trying to reach this one with hostname, this name will be resolved by dnsmasq and return with proper ip.

let me do some investigations and hopefully come up with a PR

[typhoonzero]

I think this may be a minor enhancement. We have many cases that the cluster is already there and we only need to bootstrap kubernetes and ceph etc.

[putcn]

I agree, and this is why there is a fabric version I guess?
here are my 2 cents: if the user chose to use the original version of the sextant, that means he would like to rely on sextant's dhcp which means he has a very basic and bare network env. in this case, only dhcp is not sufficient to bring up a cluster e2e. that's why I would like to add a name service to the original version.
but you are right, in most of the case, cluster would already have the dhcp, name service, etc. ready in the network, this update would not mean that much for real users in production. I just want to close the ring for original version user, so that they wouldn't be stuck somewhere.

[typhoonzero]

Is there some quick fix for this issue? I would prefer a quick fix for this:

1. dnsmasq does not suit large cluster deployment due to lack of HA hand poor performance. If user choose to use dnsmasq we definitely need a reliable name server.
2. for clusters that already have name service and OS installed, need a quick method to deploy k8s for it.

[putcn]

i've worked out a working example in my virtual box deployment. Hopefully, I can create a PR by end of today for this update.
I agree with your comments regarding dnsmasq's reliability and performance issue. Since the original version is only targeting experimenting or really small scale cluster, I think we are at least good at this point with the fix I'm working on, but we do need to document the limitations in readme.
I don't quite understand your 2nd comment, don't we already have fiber version for the case you mentioned?

[putcn]

ah, I see what you saying, you mean quicky deploy k8s in clusters which already have os installed?
I tried kubeadm, pretty neat. the only downside is: it's still in early alpha, not sure if that can be used in production.

[putcn]

I'm creating another issue regarding name service alone and keep this one open until the k8s services are verified e2e