Tackling Asynchronous Challenges

Today, my primary focus was on the integration of Epoch and Catacloud, specifically around adding events. This work mainly took place in the afternoon and evening, as the morning and part of the afternoon were dedicated to childcare. I ended up working until 3 AM.

File and Job Linking in Epoc

The integration with Epoch and Catacloud revolved around linking files and jobs. I implemented two types of events: link file and unlink file. When files are removed from a job, an unlink command is issued, and when files are added, a link command is issued.

Addressing Asynchronous Event Propagation

I encountered an asynchronous problem where the job file link (or job file projection) was not immediately updated after events were sent to the job aggregate. This meant the preview displayed old files, which was frustrating. To resolve this temporarily, I switched the event bus from the Postgres event bus to an in-memory event bus. This ensures instantaneous propagation, projecting the changes as soon as the event is emitted, without the delay of going through Postgres.

Lessons from Iris: Local Handlers for Event Buses

This issue reminded me of a similar problem I solved with Iris, a library I developed a few years ago for event-driven systems (prior to adopting event sourcing). In Iris, if the event emitter and subscriber were in the same process, the handler would be called directly in addition to emitting the event to the bus for other subscribers. This significantly reduced latency. I believe a similar functionality will need to be added to the Epoch Postgres event bus, and potentially to future Kafka or RabbitMQ event buses, to reduce latency for local subscribers. While this isn't ideal, as files attached to a job feel like part of the job projection, it resolves the immediate problem.

Enhancing Epoch's Persistence Layer

Another area for improvement in Epoch is its persistence layer. Currently, the get method has a hard-coded query, and the persist method only handles single entity saves. Ideally, the persist interface should allow saving multiple entities simultaneously. For instance, in our scenario, we have a job table and a job_file intermediary table. The job aggregate should be able to persist both the job entity and multiple job_file entities at the same time, logically representing a job with an array of files. This improvement to the Epoch library is necessary to better support these types of projections.

Progress and Future: File Projections and Billing

Despite these challenges, file linking is now working, and I've resolved several other related issues in both the UI and backend. I've also started migrating files to projections. This is a crucial step that will enable me to measure the volume of storage a specific user is consuming, which will be used for billing purposes (e.g., gigabytes per hour). My next steps involve adding an aggregate for this and then a projection to calculate "gigabytes per unit of time" as the basis for charging.

This was a productive day, and I'm making steady progress towards the billing infrastructure.