Kube-proxy in modalità iptables non funziona

hoKube-proxy in modalità iptables non funziona

kubernetes: v.1.1.1
iptables v1.4.21
kernel: 4.2.0-18-generic che vengono con Ubuntu scaltro
networking è fatto tramite L2 VLAN terminato interruttore on
nessun provider cloud

quello che faccio

sto sperimentando modalità di iptables per Kube-proxy. L'ho abilitato con argomento --proxy_mode=iptables. Sembra che qualche regola manca:

iptables -t nat -nvL 

Chain PREROUTING (policy ACCEPT 8 packets, 459 bytes) 
pkts bytes target  prot opt in  out  source    destination   
2116 120K KUBE-SERVICES all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kubernetes service portals */ 

Chain INPUT (policy ACCEPT 0 packets, 0 bytes) 
pkts bytes target  prot opt in  out  source    destination   

Chain OUTPUT (policy ACCEPT 2 packets, 120 bytes) 
pkts bytes target  prot opt in  out  source    destination   
    718 45203 KUBE-SERVICES all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kubernetes service portals */ 

Chain POSTROUTING (policy ACCEPT 5 packets, 339 bytes) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MASQUERADE all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kubernetes service traffic requiring SNAT */ mark match 0x4d415351 

Chain KUBE-NODEPORTS (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/docker-registry-fe:tcp */ tcp dpt:31195 MARK set 0x4d415351 
    0  0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/docker-registry-fe:tcp */ tcp dpt:31195 
    0  0 MARK  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* mngbox/jumpbox:ssh */ tcp dpt:30873 MARK set 0x4d415351 
    0  0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* mngbox/jumpbox:ssh */ tcp dpt:30873 

Chain KUBE-SEP-5IXMK7UWPGVTWOJ7 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.160.8   0.0.0.0/0   /* mngbox/jumpbox:ssh */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* mngbox/jumpbox:ssh */ tcp to:10.116.160.8:22 

Chain KUBE-SEP-BNPLX5HQYOZINWEQ (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.161.6   0.0.0.0/0   /* kube-system/monitoring-influxdb:api */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-influxdb:api */ tcp to:10.116.161.6:8086 

Chain KUBE-SEP-CJMHKLXPTJLTE3OP (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.254.2   0.0.0.0/0   /* default/kubernetes: */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/kubernetes: */ tcp to:10.116.254.2:6443 


Chain KUBE-SEP-GSM3BZTEXEBWDXPN (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.160.7   0.0.0.0/0   /* kube-system/kube-dns:dns */ MARK set 0x4d415351 
    0  0 DNAT  udp -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/kube-dns:dns */ udp to:10.116.160.7:53 


Chain KUBE-SEP-OAYOAJINXRPUQDA3 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.160.7   0.0.0.0/0   /* kube-system/kube-dns:dns-tcp */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/kube-dns:dns-tcp */ tcp to:10.116.160.7:53 

Chain KUBE-SEP-PJJZDQNXDGWM7MU6 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.160.5   0.0.0.0/0   /* default/docker-registry-fe:tcp */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/docker-registry-fe:tcp */ tcp to:10.116.160.5:443 

Chain KUBE-SEP-RWODGLKOVWXGOHUR (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.161.6   0.0.0.0/0   /* kube-system/monitoring-influxdb:http */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-influxdb:http */ tcp to:10.116.161.6:8083 

Chain KUBE-SEP-WE3Z7KMHA6KPJWKK (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.161.6   0.0.0.0/0   /* kube-system/monitoring-grafana: */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-grafana: */ tcp to:10.116.161.6:8080 

Chain KUBE-SEP-YBQVM4LA4YMMZIWH (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.161.3   0.0.0.0/0   /* kube-system/monitoring-heapster: */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-heapster: */ tcp to:10.116.161.3:8082 

Chain KUBE-SEP-YMZS7BLP4Y6MWTX5 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.160.9   0.0.0.0/0   /* infra/docker-registry-backend:docker-registry-backend */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* infra/docker-registry-backend:docker-registry-backend */ tcp to:10.116.160.9:5000 

Chain KUBE-SEP-ZDOOYAKDERKR43R3 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 MARK  all -- *  *  10.116.160.10  0.0.0.0/0   /* default/kibana-logging: */ MARK set 0x4d415351 
    0  0 DNAT  tcp -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/kibana-logging: */ tcp to:10.116.160.10:5601 

Chain KUBE-SERVICES (2 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SVC-JRXTEHDDTAFMSEAS tcp -- *  *  0.0.0.0/0   10.116.0.48   /* kube-system/monitoring-grafana: cluster IP */ tcp dpt:80 
    0  0 KUBE-SVC-CK6HVV5A27TDFNIA tcp -- *  *  0.0.0.0/0   10.116.0.188   /* kube-system/monitoring-influxdb:api cluster IP */ tcp dpt:8086 
    0  0 KUBE-SVC-DKEW3YDJFV3YJLS2 tcp -- *  *  0.0.0.0/0   10.116.0.6   /* infra/docker-registry-backend:docker-registry-backend cluster IP */ tcp dpt:5000 
    0  0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- *  *  0.0.0.0/0   10.116.0.2   /* kube-system/kube-dns:dns cluster IP */ udp dpt:53 
    0  0 KUBE-SVC-WEHLQ23XZWSA5ZX3 tcp -- *  *  0.0.0.0/0   10.116.0.188   /* kube-system/monitoring-influxdb:http cluster IP */ tcp dpt:8083 
    0  0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- *  *  0.0.0.0/0   10.116.1.142   /* default/docker-registry-fe:tcp cluster IP */ tcp dpt:443 
    0  0 MARK  tcp -- *  *  0.0.0.0/0   10.116.254.3   /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 MARK set 0x4d415351 
    0  0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- *  *  0.0.0.0/0   10.116.254.3   /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL 
    0  0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- *  *  0.0.0.0/0   10.116.254.3   /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 ADDRTYPE match dst-type LOCAL 
    0  0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- *  *  0.0.0.0/0   10.116.0.2   /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53 
    0  0 KUBE-SVC-7IHGTXJ4CF2KVXJZ tcp -- *  *  0.0.0.0/0   10.116.1.126   /* kube-system/monitoring-heapster: cluster IP */ tcp dpt:80 
    0  0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- *  *  0.0.0.0/0   10.116.1.175   /* mngbox/jumpbox:ssh cluster IP */ tcp dpt:2345 
    0  0 MARK  tcp -- *  *  0.0.0.0/0   10.116.254.3   /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 MARK set 0x4d415351 
    0  0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- *  *  0.0.0.0/0   10.116.254.3   /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL 
    0  0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- *  *  0.0.0.0/0   10.116.254.3   /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 ADDRTYPE match dst-type LOCAL 
    0  0 KUBE-SVC-6N4SJQIF3IX3FORG tcp -- *  *  0.0.0.0/0   10.116.0.1   /* default/kubernetes: cluster IP */ tcp dpt:443 
    0  0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- *  *  0.0.0.0/0   10.116.0.233   /* default/kibana-logging: cluster IP */ tcp dpt:8888 
    0  0 MARK  tcp -- *  *  0.0.0.0/0   10.116.254.3   /* default/kibana-logging: external IP */ tcp dpt:8888 MARK set 0x4d415351 
    0  0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- *  *  0.0.0.0/0   10.116.254.3   /* default/kibana-logging: external IP */ tcp dpt:8888 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL 
    0  0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- *  *  0.0.0.0/0   10.116.254.3   /* default/kibana-logging: external IP */ tcp dpt:8888 ADDRTYPE match dst-type LOCAL 
    0  0 KUBE-NODEPORTS all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL 

Chain KUBE-SVC-6N4SJQIF3IX3FORG (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-CJMHKLXPTJLTE3OP all -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/kubernetes: */ 

Chain KUBE-SVC-7IHGTXJ4CF2KVXJZ (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-YBQVM4LA4YMMZIWH all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-heapster: */ 

Chain KUBE-SVC-B6ZEWWY2BII6JG2L (3 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-ZDOOYAKDERKR43R3 all -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/kibana-logging: */ 

Chain KUBE-SVC-CK6HVV5A27TDFNIA (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-BNPLX5HQYOZINWEQ all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-influxdb:api */ 

Chain KUBE-SVC-DKEW3YDJFV3YJLS2 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-YMZS7BLP4Y6MWTX5 all -- *  *  0.0.0.0/0   0.0.0.0/0   /* infra/docker-registry-backend:docker-registry-backend */ 

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-OAYOAJINXRPUQDA3 all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/kube-dns:dns-tcp */ 

Chain KUBE-SVC-GLKZVFIDXOFHLJLC (4 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-5IXMK7UWPGVTWOJ7 all -- *  *  0.0.0.0/0   0.0.0.0/0   /* mngbox/jumpbox:ssh */ 

Chain KUBE-SVC-JRXTEHDDTAFMSEAS (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-WE3Z7KMHA6KPJWKK all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-grafana: */ 

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-GSM3BZTEXEBWDXPN all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/kube-dns:dns */ 

Chain KUBE-SVC-WEHLQ23XZWSA5ZX3 (1 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-RWODGLKOVWXGOHUR all -- *  *  0.0.0.0/0   0.0.0.0/0   /* kube-system/monitoring-influxdb:http */ 

Chain KUBE-SVC-XZFGDLM7GMJHZHOY (4 references) 
pkts bytes target  prot opt in  out  source    destination   
    0  0 KUBE-SEP-PJJZDQNXDGWM7MU6 all -- *  *  0.0.0.0/0   0.0.0.0/0   /* default/docker-registry-fe:tcp */

Quando faccio richiesta al servizio IP, nel mio caso si tratta di 10.116.0.2 ho ottenuto un errore

;; connection timed out; no servers could be reached

mentre quando faccio richiesta al 10,116. 160.7 server sta funzionando bene. Posso vedere che il traffico non è diretto alle regole del proxy Kube, quindi c'è probabilmente qualcosa che manca.

ho apprezzare molto qualsiasi suggerimento su regola

manca

EDIT Ive ha aggiornato la mia richiesta iniziale con le informazioni richieste dalla thokin mancante, indicò il modo veramente buono per eseguire il debug delle regole di iptables per Kube-proxy, e ho potuto identificare il mio problema con:

for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.160.7 -j LOG --log-prefix "[email protected]$c: "; done 
for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "[email protected]$c: "; done

seguenti comandi Poi ho eseguita: # nslookup kubernetes.default.svc.psc01.cluster 10.116.160.7 Server: 10.116.160.7 012.351.641,061 milaIndirizzo: 10.116.160.7 # 53

Name: kubernetes.default.svc.psc01.cluster 
Address: 10.116.0.1 

# nslookup kubernetes.default.svc.psc01.cluster 10.116.0.2 
;; connection timed out; no servers could be reached

Come risultato ho diverso indirizzo "fonte" e l'interfaccia in uscita:

[701768.263847] [email protected]: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=12436 PROTO=UDP SPT=54501 DPT=53 LEN=62 
[702620.454211] [email protected]: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62 
[702620.454224] [email protected]: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62 
[702626.318258] [email protected]: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318263] [email protected]: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318266] [email protected]: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318270] [email protected]: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318284] [email protected]: IN= OUT=docker0 SRC=10.116.250.252 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62

Così, con l'aggiunta del percorso

ip route add 10.116.0.0/23 dev docker0

Ora funziona bene!

fonte

2015-11-30 onorua

Per il futuro, i risultati di iptables-save sono molto più facili da leggere (per me comunque).

Non vedo niente mancante qui.

KUBE-SERVICES trappole 10.116.0.2 porte 53/UDP e lo passa al KUBE-SVC-TCOU7JCQXEZGVUNU

KUBE-SVC-TCOU7JCQXEZGVUNU ha solo un endpoint così salta KUBE-SEP-GSM3BZTEXEBWDXPN

KUBE-SEP-GSM3BZTEXEBWDXPN DNATs a 10.116.160.7 port 53/UDP

Se si asserisce che 10.116.160.7 funziona mentre 10.116.0.2 non lo fa, questo è davvero strano. Suggerisce che le regole di iptables non si attivano affatto. Stai provando dal nodo stesso o da un contenitore?

Che rete stai usando? L3 (sottopunto?) Flanella? OVS? Qualcos'altro?

Quale provider cloud (se presente)?

Primo passo per eseguire il debug: eseguire: for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "[email protected]$c: "; done

che registrerà tutti i pacchetti che iptables vede al vostro servizio IP. Ora guarda dmesg.

fonte

2015-11-30 21:37:31

grazie, i tuoi messaggi di debug iptables mi hanno segnalato il fatto che ho avuto una route mancante – onorua

Potrei avere una domanda correlata [qui] (http://stackoverflow.com/questions/43485770/route-not-working-in -kubernetes-con-calico). Potresti aiutarmi? – JasonW

Kube-proxy in modalità iptables non funziona

risposta

Problemi correlati