BlueDog's Home

In today's widespread application of cloud-native architecture, the illegal external network behavior of containers in Kubernetes (K8s) clusters has become an important focus of offensive and defensive confrontation. This article will deeply analyze how to accurately locate the source of external connections and visualize real-time alerts, and build proactive prevention mechanisms from emergency investigation, mandatory isolation to policy protection.

I. Core Process of Emergency Investigation for External Connections#

When a Pod in Kubernetes is deleted, if there is no controller (such as Deployment, ReplicaSet, etc.) that deleted the Pod, K8s will automatically recreate the Pod based on the defined replica count. The following are the steps to investigate and resolve this issue:

Check Pod Status#

kubectl get pods -A -owide

This command lists all Pods across all namespaces and displays their detailed information (including IP addresses, statuses, etc.).

Delete Specified Pod#

kubectl delete pod <pod-name> -n <namespace>

This command will delete the specified Pod. Note that if the controller has defined a replica count, the Pod will be automatically rebuilt.

Delete Replica Controller#

To completely stop the Pod from being rebuilt, you need to delete its managing replica controller (such as Deployment, ReplicaSet):

kubectl get deployment -A
kubectl delete deployment <deployment-name> -n <namespace>

This will delete the specified Deployment, and the controller will no longer schedule new Pods.

Force Delete Pod#

If a Pod is stuck or cannot be deleted normally, you can use the following command to force delete it:

kubectl delete pod <name> --grace-period=0 --force -n <namespace>

--grace-period=0 will immediately terminate the Pod, while --force will skip any graceful termination steps.

II. Configure NetworkPolicy to Prohibit External Access#

NetworkPolicy is a mechanism provided by Kubernetes to control the inbound (Ingress) and outbound (Egress) traffic of Pods. By default, all traffic in Kubernetes is allowed, but by configuring NetworkPolicy, you can restrict the external network behavior of containers.

Configure NetworkPolicy to Restrict External Connections#

The following is an example configuration that prohibits all containers from accessing external networks:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-egress-to-external
  namespace: default
spec:
  podSelector: {}  # Matches all Pods
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8  # Only allows communication with internal networks

In the above configuration, the egress section restricts the Pod to communicate only with IP addresses within the 10.0.0.0/8 subnet. This prevents the Pod from making requests to external networks.

Configuration Details and Verification#

• Default Deny: If the NetworkPolicy has no matching rules, Kubernetes will deny all traffic by default.
• Inbound Traffic Restrictions: Similarly, you can restrict Pods from accepting traffic from external sources.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-ingress
  namespace: default
spec:
  podSelector: {}  # Matches all Pods
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 192.168.1.0/24  # Only allows traffic from 192.168.1.0/24

You can use the kubectl describe networkpolicy command to view the configured NetworkPolicy.

III. Build Real-Time Alerting Chain with Falco + Prometheus + Grafana#

Introduction to Falco#

Falco is a CNCF graduated project designed as a runtime threat detection system for container environments, mainly used for real-time monitoring of system calls (via eBPF or kernel modules) and detecting abnormal behaviors (such as privilege escalation, external connections, abnormal executions) based on predefined rules, thus achieving runtime security monitoring and threat detection in cloud-native environments.

Configure Falco Monitoring#

The following rules detect outbound connections initiated by containers, excluding local network segments (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), and log events connecting to external IPs.

- rule: Outbound Connection in Container
  desc: Detects outbound network connection from container to external IP
  condition: >
    evt.type = connect and
    container and
    not fd.sip in (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
  output: >
    Container outbound connection to external IP: (command=%proc.cmdline connection=%fd.name user=%user.name)
  priority: WARNING
  tags: [network, outbound]

⚠️ You can combine fd.sip to match illegal target IPs for precise identification of cross-boundary communication.

Integrate Prometheus + Grafana#

1. Deploy Falco-exporter to output metrics to Prometheus.

2. Customize Grafana Dashboard:

   • Display the frequency of rule triggers.
   • Alert IP/process/container associations.
   • Visualize external connection trends in the cluster.

Trigger Alert Mechanism#

Configure falcosidekick to push alerts to Slack/DingTalk, Webhook (automatically isolate in conjunction with SOAR), Prometheus AlertManager.

IV. Use OPA Gatekeeper to Restrict Container Configuration Privileges#

Open Policy Agent (OPA) is a general-purpose policy engine that can be used to control the creation and management of Kubernetes resources. Gatekeeper is a Kubernetes controller for OPA that can enforce policy management within Kubernetes clusters. Through Gatekeeper, users can define and enforce various security policies.

Scenario: Prohibit the Use of hostNetwork=true When Deploying Containers (Bypassing Network Policies)#

ConstraintTemplate Definition#

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sdenyhostnetwork
spec:
  crd:
    spec:
      names:
        kind: K8sDenyHostNetwork
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdenyhostnetwork

        violation[{"msg": msg}] {
          input.review.object.spec.hostNetwork == true
          msg := "hostNetwork usage is forbidden"
        }

Constraint Binding#

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDenyHostNetwork
metadata:
  name: deny-host-network
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]

After deploying this policy, any Pod attempting to enable hostNetwork: true will be rejected by the Admission Webhook.

Examples of Similar Policy Applications#

V. Additional Strategies for Further Strengthening#

Strengthen Boundary Security#

• Restrict egress IP: Use Calico/Cilium to define finer-grained network policies;
• PodSecurityAdmission: Prevent running privileged containers, allow specific user UIDs to run;
• KubeArmor / Tetragon: Process-level runtime behavior detection (combined with LSM / eBPF);
• Service Mesh ACL: Fine-grained traffic access control based on Istio;
• DNS Firewall: Deploy DNS proxy in front of CoreDNS to intercept malicious domains;

Integrate Audit Platforms#

• Cluster behavior auditing: Such as kube-audit-log in conjunction with ELK;
• Network behavior archiving: Such as integrating Suricata for packet analysis;
• Asset ownership labeling: Record department/system/application in Pod labels for traceability;

Conclusion#

In the face of complex and changing external connection risks in Kubernetes, a single tool or method is insufficient to address global issues. It is recommended to integrate "post-event investigation response" with "pre-event policy prevention" to form a closed loop of "detection, blocking, visibility, traceability, response" through the following methods, enhancing the resilience and pressure resistance of the cluster itself.

• PodSecurityPolicy: Restrict the use of privileged modes for containers, enforce image signing, limit the users and groups under which containers run;
• Service Mesh: In environments like Istio, strengthen access control at the network layer and implement fine-grained access control policies;
• MageScanning: Use tools like Clair / Trivy to scan container images for vulnerabilities, preventing the deployment of vulnerable images.