Podłączenie producera kafki z innego kontenera

Mam problem z wrzucaniem producerem wiadomości na topic kafki. Sytuacja wygląda tak (aplikacje spring boot):

processor-service i kafka są w jednym docker-compose. Consumer będący w processor-service udanie subskrybuje topic.
client-service w drugim docker-compose nie może wrzucić message na topic. Otrzymany błąd to:

Kopiuj

2023-03-28T18:29:59.611Z  INFO 1 --- [   scheduling-1] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
2023-03-28T18:29:59.673655000Z 2023-03-28T18:29:59.672Z  INFO 1 --- [   scheduling-1] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.3.2
2023-03-28T18:29:59.674622100Z 2023-03-28T18:29:59.673Z  INFO 1 --- [   scheduling-1] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: b66af662e61082cb
2023-03-28T18:29:59.675266800Z 2023-03-28T18:29:59.675Z  INFO 1 --- [   scheduling-1] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1680028199670
2023-03-28T18:29:59.999613100Z 2023-03-28T18:29:59.999Z  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Node -1 disconnected.
2023-03-28T18:30:00.009419500Z 2023-03-28T18:30:00.005Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/127.0.0.1:9092) could not be established. Broker may not be available.
2023-03-28T18:30:00.009451000Z 2023-03-28T18:30:00.006Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected

W processor-service wygląda ok:

Kopiuj

2023-03-28T18:28:23.894Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.3.2
2023-03-28T18:28:23.895381800Z 2023-03-28T18:28:23.895Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: b66af662e61082cb
2023-03-28T18:28:23.895458800Z 2023-03-28T18:28:23.895Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1680028103893
2023-03-28T18:28:23.899088100Z 2023-03-28T18:28:23.898Z  INFO 1 --- [           main] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-firstGroup-1, groupId=firstGroup] Subscribed to topic(s): topic-1
2023-03-28T18:28:23.948411700Z 2023-03-28T18:28:23.948Z  INFO 1 --- [           main] c.m.p.ProcessorServiceApplication        : Started ProcessorServiceApplication in 8.57 seconds (process running for 10.654)
...
2023-03-28T18:28:27.102Z  INFO 1 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : firstGroup: partitions assigned: [topic-1-0]

oraz oba docker-compose

Kopiuj

version: '2'

services:
  app:
    image: 'processor-service-image'
    ports:
      - "8080:8080"
    build:
      context: .
    container_name: processor-service
    depends_on:
      - db
    environment:
      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/processor-service-database
      - SPRING_DATASOURCE_USERNAME=docker_user
      - SPRING_DATASOURCE_PASSWORD=pass
      - SPRING_JPA_HIBERNATE_DDL_AUTO=create-drop

  db:
    image: 'postgres:latest'
    container_name: postgres-container
    environment:
      - POSTGRES_USER=docker_user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=processor-service-database
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_HOST: localhost
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ADVERTISED_HOST_PORT: 9092

Kopiuj

version: '2'

services:
  app:
    image: 'client-service-image'
    ports:
      - "8081:8081"
    build:
      context: .
    container_name: client-service
    environment:
      SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092

Kopiuj

λ docker ps -a
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                  PORTS                                                NAMES
41f146c81cb7        processor-service-image   "java -jar /processo…"   24 seconds ago      Up 22 seconds           0.0.0.0:8080->8080/tcp                               processor-service
589800857a7f        wurstmeister/kafka        "start-kafka.sh"         27 seconds ago      Up 24 seconds           0.0.0.0:9092->9092/tcp                               kafka
7d66e0804386        wurstmeister/zookeeper    "/bin/sh -c '/usr/sb…"   27 seconds ago      Up 24 seconds           22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   zookeeper
d06af49b1d37        postgres:latest           "docker-entrypoint.s…"   27 seconds ago      Up 24 seconds           5432/tcp                                             postgres-container
54299e81a0c1        client-service-image      "java -jar /client-s…"   6 minutes ago       Up 6 minutes            0.0.0.0:8081->8081/tcp                               client-service

Serwis client-service mogę odpalić w kontenerze i lokalnie, ale w obu przypadkach nie działa. Ktoś wie jak to można rozwiązać?

Co znaczy nie działa?

Znalazłem rozwiązanie. Okazało się, że każdy docker compose tworzy swój network. Co ciekawe nie mogłem ustawić customowego dla obu bo i tak oba compose tworzyły swoje.
Dodałem więc kontenery z drugiego docker-compose do network tego pierwszego czyli:

Poprzez docker network ls sprawdziłem jaki network mają kontenery z pierwszego compose. Był to: processor-service_network-1.
Dodałem do drugiego compose:

Kopiuj

networks:
  processor-service_network-1:
    external: true

a więc cały compose wygląda tak:

Kopiuj

version: '2'

services:
  app:
    image: 'client-service-image'
    ports:
      - "8081:8081"
    build:
      context: .
    container_name: client-service
    networks:
      - processor-service_network-1
    environment:
      SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092

networks:
  processor-service_network-1:
    external: true