Getting [Error: consul: kv.get: connect ECONNREFUSED]

[ogavrisevs]

We are trying to deploy git2consul next to consul in our k8s cluster unfortunately no matter what we try its not working.

Here is our settup :

1. k8s 1.6 deployed by kops on aws
2. consul (docker image : consul:1.2.0) single node container
3. git2consule from ( docker image : cimpress/git2consul:0.12.13)

Error we are getting is :

 { [Error: consul: kv.get: connect ECONNREFUSED]
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect' } 

We removed every thing from consul that can impact connection :

1. Encryption GOSSIP_ENCRYPTION_KEY
2. All related to ACL acl_default_policy / etc.
3. Datacenter datacenter.
4. TLS.

On git2consul side we try to :

1. We use IP instead of DNS name.
2. We try arg options instead of environment variables.
3. Even removed --config-file to test is its not related to git repo config.

Nothing helped and we still getting that error.
I also checked connection from other pods, i am able to connect from side container by consul agent :

apiVersion: v1
kind: Pod
metadata:
  name: consul-tets
spec:
  containers:
  - name: consul
    image: consul:1.2.0
    args:
      - "agent"
      - "-advertise=$(POD_IP)"
      - "-bind=0.0.0.0"
      - "-retry-join=consul"
    env:
      - name: POD_IP
        valueFrom:
          fieldRef:
            fieldPath: status.podIP

and from cli

./consul join  100.96.2.82 
./consul members 
./consul kv get t1

Here is our consul config:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  labels:
    app: consul
    role: server
spec:
  selector:
    matchLabels:
      app: consul
      role: server
  serviceName: consul
  replicas: 1
  template:
    metadata:
      labels:
        app: consul
        role: server
    spec:
      securityContext:
        fsGroup: 1000
      containers:
      - name: consul
        image: consul:1.2.0
        imagePullPolicy: IfNotPresent
        env:
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - "agent"
        - "-advertise=$(POD_IP)"
        - "-bind=0.0.0.0"
        - "-bootstrap-expect=1"
        - "-client=0.0.0.0"
        - "-config-file=/consul/configs/server_config.json"
        - "-data-dir=/consul/data"
        - "-domain=cluster.local"
        - "-server"
        - "-ui"
        - "-disable-host-node-id"
        volumeMounts:
        - name: consul-config
          mountPath: /consul/configs
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - consul leave
        ports:
          - containerPort: 8500
            name: ui-port
          - containerPort: 8400
            name: alt-port
          - containerPort: 53
            name: udp-port
          - containerPort: 8443
            name: https-port
          - containerPort: 8080
            name: http-port
          - containerPort: 8301
            name: serflan
          - containerPort: 8302
            name: serfwan
          - containerPort: 8600
            name: consuldns
          - containerPort: 8300
            name: server
      volumes:
        - name: consul-config
          configMap:
            name: consul-config

and

 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: consul-config
 data:
   server_config.json: |-
     {
       "client_addr": "0.0.0.0",
       "data_dir": "/consul/data",
       "bootstrap_expect": 1,
       "server": true,
       "ui": true,
       "enable_debug": true,
       "log_level": "debug",
       "dns_config": {
         "enable_truncate": true,
         "service_ttl": {
           "*": "5s"
         }
       },
       "rejoin_after_leave": true,
       "raft_protocol": 3,
       "disable_update_check": true,
       "retry_interval" : "10s",
       "verify_incoming": false,
       "verify_outgoing": false,
       "verify_server_hostname": false
     }

Git2consul config :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: git2consul
  labels:
    app: git2consul
spec:
  replicas: 1
  selector:
    matchLabels:
      app: git2consul
  template:
    metadata:
      labels:
        app: git2consul
    spec:
      containers:
      - name: git2consul
        image: cimpress/git2consul:0.12.13
        args:
          - "--config-file=/etc/git2consul.d/config.json --endpoint  100.96.2.88 --port 8500"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8888
        resources: {}
        volumeMounts:
          - name: git2consul-config
            mountPath: "/etc/git2consul.d/config.json"
      restartPolicy: Always
      volumes:
        - name: git2consul-config
          configMap:
            name: git2consul-config
            items:
            - key: git2consul_config.json
              path: consul/config/git2consul_config.json

and

apiVersion: v1
kind: ConfigMap
metadata:
  name: git2consul-config
data:
  git2consul_config.json: |-
    {
      "version": "1.0",
      "local_store": "/var/lib/git2consul_cache",
      "repos": [
        {
          "name": "config-server-repo",
          "expand_keys": true,
          "url": "git@bitbucket.org:xxx/config-server-repo.git",
          "branches": [
            "prod"
          ],
          "hooks": [
            {
               "level": "debug",
              "type": "polling",
              "interval": "1"
            }
          ]
        }
      ]
    }

Hi @ogavrisevs,
What was the solution here ?