zilla stops sending sse updates after a couple days

[pnikhil610]

Describe the bug
My kafka topic receives about 10 million+ events per day. Each message has key and there are about 50k unique keys. I have zilla sse endpoint which filters based on key. After about a day, it makes only about sse 20k sse connections. Then sometime in day 2 or later, it stops sending live updates to sse client. That is, it freezes somehow. It just show one entries. Even though additional entries are pushed, I don't get them. I'm not sure if it's cache or resource issue.

To Reproduce
Steps to reproduce the behavior:

1. Use similar config as mine
2. Send lots of events in kafka topic
3. Create a several thousand sse connection
4. At some point, you'll see zilla stops sending updates

Expected behavior
Zilla should repeatedly clean cache to outcome if it's memory issue.

Screenshots
If applicable, add screenshots to help explain your problem.

Zilla Environment:
container and resources look like

      containers:
       - name: zilla
         image: ghcr.io/aklivity/zilla:0.9.101
         args:
           - start
           - '-v'
           - '-e'
           - '-Pzilla.engine.worker.capacity=128'
         env:
           - name: ZILLA_VERSION
             value: 0.9.101
         resources:
           limits:
             cpu: '2'
             memory: 2G
           requests:
             cpu: '2'
             memory: 2G
             ```

**Attach the `zilla.yaml` config file:**

My config looks like
```yaml
name: zilla-ohlc-prod-1440min
   telemetry:
     exporters:
       stdout_log_exporter:
         type: stdout
   bindings:
     north_tcp_server:
       type: tcp
       kind: server
       options:
         host: 0.0.0.0
         port:
           - 7114
       routes:
           - when:
               - port: 7114
             exit: north_http_server
     north_http_server:
       type: http
       kind: server
       options:
         access-control:
           policy: cross-origin
       routes:
         - when:
             - headers:
                 :scheme: http
                 :path: /events/*
             - headers:
                 :scheme: http
                 :path: /events
           exit: north_sse_server
     north_sse_server:
       type: sse
       kind: server
       exit: north_sse_kafka_mapping
     north_sse_kafka_mapping:
       type: sse-kafka
       kind: proxy
       routes:
         - when:
             - path: /events/key/ohlc_1440/{id}
           with:
             topic: ohlc_1440
             filters:
               - key: ${params.id}
           exit: north_kafka_cache_client
     north_kafka_cache_client:
       type: kafka
       kind: cache_client
       exit: south_kafka_cache_server
     south_kafka_cache_server:
       type: kafka
       kind: cache_server
       options:
         bootstrap: 
           - ohlc_1440
         topics:
           - name: ohlc_1440
             defaultOffset: live
       exit: south_kafka_client
     south_kafka_client:
       type: kafka
       kind: client
       exit: south_tcp_client
     south_tcp_client:
       type: tcp
       kind: client
       options:
         host: broker
         port: 9092
       routes:
         - when:
             - cidr: 0.0.0.0/0

Kafka Environment:
Confluent Kafka

Client Environment:
Chrome browser latest or any other browser

[ankitk-me]

Hello @pnikhil610 , I tried to replicate this issue locally by triggering approx 10+ millions messages/24 hours with 50,000 unique keys. And use SSE Listener to subscribe to unique keys.

After running this setup for approx ~40 hours, I can still observe SSE Listener containers able to consume latest update for the unique keys.

I have uploaded my replication recipe in this Git Repo. Can you please review this and share anything that might help us reproduce the issue.

[nikhil-cli]

can you please tell how does your ingress look like?
and other k8 and zilla settings
also if possible please share the zilla.yaml.

it will be helpful to go over them to identify issue in my setup. thanks!

[ankitk-me]

@nikhil-cli Refer to this Git Repo for information on the setup.

[jfallows]

Closing due to inactivity and unable to reproduce.