Can't query simple Redpanda topic?

[danthegoodman1]

I am testing materialize and redpanda over docker and am running into some issues. When I try to query from Materialize, I get no rows, whether I produce records before or after creating the source, materialized view, etc.

Setup:

docker run -d --pull=always --name=redpanda-1 --rm \
-p 9092:9092 \
-p 9644:9644 \
docker.vectorized.io/vectorized/redpanda:latest \
redpanda start \
--overprovisioned \
--smp 1  \
--memory 1G \
--reserve-memory 0M \
--node-id 0 \
--check=false \
--set redpanda.enable_transactions=true \
--set redpanda.enable_idempotence=true

docker run -p 127.0.0.1:6875:6875 -d materialize/materialized:v0.12.0 --workers 1

And a simple go program writing some messages to a topic:

package main

import (
	"context"
	"encoding/json"
	"log"
	"math/rand"
	"time"

	"github.com/segmentio/kafka-go"
)

var (
	Writer *kafka.Writer
)

func main() {
	rand.Seed(time.Now().UnixNano())
	log.Println("Starting Materilize and Redpanda integration test")
	topic := "my-topic"

	Writer = &kafka.Writer{
		Addr:         kafka.TCP("localhost:9092"),
		Topic:        topic,
		Balancer:     &kafka.LeastBytes{},
		BatchTimeout: time.Millisecond * 10,
	}

	for {
		log.Println("Ticking...")
		WriteRecord()
		time.Sleep(2 * time.Second)
	}
}

// Writes a random record to Redpanda
func WriteRecord() {
	log.Println("Writing record")
	var key string
	val := map[string]interface{}{}
	r := rand.Intn(2)
	switch r == 0 {
	case true:
		// 1/3 chance
		key = "low"
	case false:
		// 2/3 chance
		key = "high"
	}
	val["num"] = r

	// JSON
	bytes, err := json.Marshal(&val)
	if err != nil {
		panic(err)
	}

	t := time.Now()
	err = Writer.WriteMessages(context.Background(), kafka.Message{
		Key:   []byte(key),
		Value: bytes,
	})
	if err != nil {
		log.Fatal(err)
	}
	log.Println("Wrote message in", time.Since(t)-(time.Millisecond*10))
}

I am then creating the following source and views such as:

# https://materialize.com/taking-streaming-analytics-further-faster-with-redpanda-materialize/
CREATE MATERIALIZED SOURCE sss
  FROM KAFKA BROKER '172.17.0.2:9092' TOPIC 'my-topic'
  FORMAT bytes
  ENVELOPE NONE;

CREATE MATERIALIZED VIEW jsonified_kafka_source AS
  SELECT CAST(data AS jsonb) AS data
  FROM (
      SELECT convert_from(data, 'utf8') AS data
      FROM sss
  );

CREATE MATERIALIZED VIEW vvv AS
        SELECT COUNT(*) as countts
        FROM jsonified_kafka_source
        ;

When every I run SELECT * FROM vvv I always have a rowcount of 0 even though I have no trouble connecting and the go program is having no issues producing. Am I missing something here?

[danthegoodman1]

This seems to have been a networking issue. If I use the following docker-compose.yml and reference redpanda:9092 it seems to work. Will wait for an official answer to confirm. Although now I can't write from my go instance due to the - --advertise-kafka-addr redpanda:9092 line (it creates the topic, but I get a write error still).

version: '3'
services:
  redpanda:
    image: docker.vectorized.io/vectorized/redpanda:v21.9.5
    command:
     - redpanda start
     - --overprovisioned
     - --smp 1
     - --memory 1G
     - --reserve-memory 0M
     - --node-id 0
     - --check=false
     - --kafka-addr 0.0.0.0:9092
     - --advertise-kafka-addr redpanda:9092
     - --pandaproxy-addr 0.0.0.0:8082
     - --advertise-pandaproxy-addr redpanda:8082
     - --set redpanda.enable_transactions=true
     - --set redpanda.enable_idempotence=true
    ports:
     - 9092:9092
     - 8081:8081
     - 8082:8082
  materialized:
    image: materialize/materialized:v0.12.0
    init: true
    command: --workers=1
    ports:
      - 6875:6875

[morsapaes]

Hey, Dan!

Using Docker Compose does make things easier networking-wise, so I'm glad you sorted that out! There are a couple of examples of end-to-end Materialize+Redpanda setups here and here, if it helps.

Are you using CREATE MATERIALIZED SOURCE for testing, so you can directly query the source? Just wanted to point out that this is not needed (or recommended) otherwise, since it'll index the entire source in memory! You can simply use CREATE SOURCE and then materialize whatever views you need on top of that. The same applies for casting and converting the original raw bytes: you should define these intermediate steps as views using CREATE VIEW. Our getting started guide has a brief step-by-step explanation on how to handle JSON source data!

[danthegoodman1]

Yeah I was throwing everything at it, definitely using non-materialized sources in production. I am surprised that materialize does not care about the fact it was unable to connect, was I just being impatient for some long time out?

[morsapaes]

I'd expect to find some sort of request timeout in the logs of materialized. Weren't there any helpful log entries, in your case?

[danthegoodman1]

I didn’t check there, I was referring to the query failing if it can’t connect but I guess that is async. Will know to check there from now on.

maybe a flag to block for connections in a future feature?