RFC stream to stream projections

[prolic]

Assumed you use one stream per aggregate, and you have streams names like:

user-edb72b9e-1896-49bf-9fda-7c8306ca34b1
user-cb8c6891-8eb3-45e3-bffb-1ab4b0ec31b1

We could make a projection from those two streams to a new event stream users - that is also an event stream but combines both streams in one.

We could make another projection from all streams like

user-{id1} ,user-{id2}, blogpost-{id1}, blogpost-{id2} to a category streams like when contains: user and blogpost - the rule would be the first part of an stream name until the first - is the category name.

Next possibility would be one stream projection per event type.

Next possibility would be one stream for all stream names.

Most of them would be internal stream names (prefixed with _) and are optional to enable (maybe a distinct package, too, relying on amqp or pthreads?).

[codeliner]

I only see this in a distinct package.
You need a lot of stuff (projection workers, process managers, ...) and you have dependencies to amqp/zeromq and/or pthreads.

pthreads only work in CLI context, so you'd need to forward all recorded events to your workers using amqp/zeromq.

[prolic]

I don't think AMQP/ZeroMQ/pthreads is necessary. We will have the HTTP API and the projection can work with this and ask for stuff it needs, instead of relying on a queue. This means also there is no need anymore to ask the backend to send streams again to a new queue created for this projector.
The projector is independent and can ask what it needs himself.

But it still probably will be a distinct repository.

[codeliner]

ok, HTTP API can work as a default, but the projection need to poll the API continuously or even use web sockets or HTTP/2 persistent connections which would require a reactPHP http server or similar to host the HTTP API.
And you produce a lot of extra load on your event store db.
The latter can be solved with master-slave set ups, though.

[prolic]

Usually you want to replay a specific projection (not all of them), therefore it's way easier to query the HTTP API for it. No need to ask the backend to create a queue and push everything there.

However I agree, that this might be the better solution, if you want to replay all of your projections.

Either way, there is a query to the database and some events are read from there.

Let's make it configurable and HTTP API is the default.

[codeliner]

Do we only talk about replay scenarios here? I thought stream-to-stream projection is a continuous process so that your category streams are eventually up-to-date?

[prolic]

yes it is, however the stream projection is a projection. you can throw it away and replay. you can decide at a later point in time that you need it and enable later.

[codeliner]

Ok yeah. I'm just asking because of the HTTP API long-polling issue. We need to think about that or play with different ideas.

[prolic]

see #213