Miquel Puig Turon

Humanity

Thu, 08 Oct 2026 13:07:43 GMT

Humanity

Tangible things that might aid other people. Possibly unbeknownst to me.

I strive to have a positive effect on humanity.

Short term, it will likely be outcomes of my work.
Long term, it might document project(s) that I build.

Self

Thu, 08 Oct 2026 13:07:43 GMT

Self

Beliefs around how I live my life.

Or maybe attempts at explaining why.

I will write about the things that are most unique to me.
It is not intended to be a guide to follow, but as a description to understand.

Welcome

Thu, 08 Oct 2026 13:07:43 GMT

Welcome

This is my personal site.
A caricature of my thoughts.

Why

I’ve lived most of my life feeling misunderstood.

Partly because my convictions differ so much from what’s considered normal.
Mostly because of my inability to express my understanding of the world in simple words.

Perfection has always been an inquietude within me, yet forever beyond reach.
By definition, perfection is immutable, and I’ll always grow. Still, some ideas are deeply rooted in me.

This is an attempt to understand.

What

I intend for this website to be a dump of my brain.

Not a blog, not a collection of anecdotes. A fully-connected network of thoughts.

Each article lives on a path, making this page the root of a tree.
To comprehend why and what is written in an article, its path and siblings matter.

This is an attempt at perfection.

Where to start

If you’re not sure which to read, three pieces stand on their own:

Pain, Suffering and God — why I build, and what it costs.
Tres mesos intentant construir software per a farmàcies — the full autopsy of a startup that didn’t work (in Catalan).
Boxing — a year of getting hit, and what it settled.

When you’re done, the whole tree is right below — broader near the root, specifics deeper down.

See how this page was built.

Fire

Thu, 08 Oct 2026 13:07:43 GMT

Fire

My thoughts building :fire:

Money

Thu, 08 Oct 2026 13:07:43 GMT

Money

I believe few companies will monopolize the market.

Money will mostly become meaningless, and almost everyone will both live in abundance and be ridiculously poor.

How money grows

For most people, money grows through salary and goes away via day to day expenses.

Salaries grow slower than assets, and this compounds.

It is very hard to jump an order of magnitude.

If someone starts from scratch, it will likely take its entire live to acquire 1M$ of assets. If another starts from 1M$ of assets, it will likely end its live with 100M$ of assets.

The math is simple, put your money in any vehicle that returns in proportion to its money: d$/dt = k$ => $(t) = $_0 * e^kt

To maintain the money difference with someone, you need to grow your money at the same rate he does.
If you add to your wealth X% MoM, in an infinite amount of time, you will be wealthier than anyone who adds less, no matter the initial amount.

Why non-asset owners won’t be rich

Given a modest start, you need to work to stay alive.

If you save a fixed amount, you might reach a point where returns from your assets are higher than your salary.
You can stop working and stop adding money, or keep a job to pay for your expenses and let money grow.
At some point, you might have enough assets that income from your job will be a rounding error and then you are rich.

Notice though, how default state for those who do not own assets is to survive vs default state for asset owners, which is to compound wealth.
This means that by default, the gap between rich and poor grows bigger.

We will reach a point where salary will not be enough to get you to own assets. (I believe we are not far from it)

Why inheritance sucks

In a world where we are slaves of abundance, assets will only be able to propagate through inheritance.

As the positions to get rich from nothing grow thinner, and needs are abundantly met, there’s no point in effort.

Why would people push themselves when there is no reward?

I think the only fair way to redistribute assets is killing inheritance.

My ideas go much further. I think education should be globally homogeneous and parent’s money should not influence it, else we are propagating wealth through education.
But that’s writing for another day.

Tres mesos intentant construir software per a farmàcies

Thu, 08 Oct 2026 13:07:43 GMT

Tres mesos intentant construir software per a farmàcies

Hem entrevistat unes 70 farmàcies, hem instal·lat el producte a 12 i hem acabat concloent que el model de negoci en que crèiem no funciona.

Per què farmàcia

En Lluís és amic meu, fill de farmacèutics, ha passat tota la vida a la farmàcia. Em va explicar els seus problemes, idees i el panorama actual del software. Jo hi vaig veure l’oportunitat d’aportar valor.

El mercat de la farmàcia

La farmàcia és un negoci de marge baix que cada cop és més difícil de defensar. Hi ha dos camins que les farmàcies estan prenent per mantenir-se competitives.

Volum i marge

Una estratègia en augment a Europa, i amb gran èxit a França, és unir la farmàcia a un grup. Els grups de compra tenen més poder de negociació que les farmàcies independents, i per tant costos d’adquisició més baixos. Més enllà dels grups de compra hi ha les “marques”, que afegeixen imatge, consultoria i eines de gestió.

Tots dos models són atractius per a la farmàcia perquè simplifiquen la gestió i ajuden a pujar el sell-out. En una situació on la gestió integral requereix cada cop més feina i el retorn és més difícil d’aconseguir, és fàcil entendre per què creixen.

Diferenciació i serveis

La farmàcia cada vegada més es posiciona com a punt de referència per fer molt més que dispensar: oferir serveis. Ortopèdia, anàlisi de paràmetres biomèdics, SPD… Això ve impulsat des de la pròpia farmàcia i des d’organitzacions governamentals, que volen reduir feina de les institucions mèdiques i convertir la farmàcia en referent sanitari.

Orkava

El farmacèutic té poc temps i ha de fer mil coses, i sovint no té els processos optimitzats: gestió de stock, substitució de genèrics, cross-selling, fidelització de pacients… Orkava neix per ajudar amb tot això. La resta és la història de com hem anat canviant d’hipòtesi per aconseguir-ho.

Què hem après

Quan vam començar, pensàvem que el farmacèutic necessitava una capa d’intel·ligència sobre l’ERP per prendre millors decisions.

Vam atacar els processos més evidents: gestió de stock, substitució de genèrics, cross-selling, fidelització de pacients, tasques internes i horaris. Totes aquestes funcionalitats aporten valor, però cap forma part del dia a dia. Es consulten quan hi ha un dubte concret, però no generen hàbit.

Ho vam confirmar amb els primers resultats: hem parlat amb unes 70 farmàcies i n’hem instal·lat 12 amb seguiment personalitzat. D’aquestes, només 4 l’usen a diari, 5 setmanalment i 3 no l’usen. Cap explota Orkava com imaginàvem.

Vam intentar reduir la fricció posant-ho tot dins d’un xat. Ha estat útil per a preguntes concretes, com analitzar una marca o entendre una baixada de marge, però no ha creat pull. El farmacèutic només obre el xat quan ja té una pregunta, i això passa poc.

Després vam fer Orkava més proactiu amb notificacions sobre impagats, stock parat, caducitats o altres incidències. Tampoc va funcionar: moltes alertes no porten a una acció clara i és difícil demostrar valor sense mesurar si l’acció s’ha fet ni quin impacte ha tingut.

El mòdul de màrqueting va seguir el mateix patró. Les farmàcies que ja feien màrqueting el van aprofitar i no volien cambiar; les que no, no van crear nous processos.

La següent hipòtesi va ser que estàvem fora del flux de treball. Si volíem formar part del dia a dia, havíem d’estar presents en el moment de la venda. Integrar-nos amb l’ERP va ser molt més difícil del que esperàvem, però ho vam aconseguir per altres vies :).

El resultat tampoc va ser suficient. Els popups semblen útils al principi, però ràpidament es converteixen en una cosa més a la pantalla. Les recomanacions centrades en el pacient i el seu historial aporten més valor que els suggeriments típics de producte, però no vam veure prou evidència que poguessin convertir-se en una part indispensable de la venda.

La conclusió és que no hem aconseguit entrar en cap workflow indispensable del farmacèutic. Hem construït una capa d’intel·ligència per a la farmàcia, però depèn que el farmacèutic decideixi consultar-la. I aquest és precisament el problema: si el producte no forma part d’un procés obligatori, acaba sent una eina que sembla útil, però que cada vegada s’utilitza menys.

Els competidors

El producte més similar a Orkava que està ben posicionat és Sisfarma. Es posicionen com el CRM de la farmàcia. La webapp és una castanya, i els farmacèutics amb qui hem parlat que el tenen no la fan servir.

Com han aconseguit entrar al mercat? Perquè en realitat són una consultoria.

NO ESTÁS SOLO: TECNOLOGÍA CON ALMA DE CONSULTORÍA

Sisfarma no es solo un software. Ponemos a tu disposición un equipo de asesores expertos en rentabilidad con los que te reunirás cada mes para analizar tu farmacia, fijar estrategias y garantizar resultados.

La consultoria funciona al món de la farmàcia, i hi ha diversos jugadors. Sisfarma ho ha fet molt bé: en construir la plataforma, amb menys dedicació i personal poden oferir una consultoria suficient.

Intelia Pharma posa un popup que et recomana productes en funció d’altres productes. Les farmàcies amb qui hem parlat que el tenen estan descontentes: és car i s’acaba ignorant. Crec que el seu enfocament és erroni, perquè parteix del producte i no de l’historial del pacient.

No hi ha cap jugador que hagi aconseguit entrar al mercat posant-se en el moment de la venda.

Menció a Blistersuite, que ha trobat un flux que el farmacèutic ha de fer per nova normativa i que no està integrat a l’ERP. El programa deixa molt a desitjar, però fa la seva funció i ha capturat el mercat (que és petit).

Oportunitats

L’oportunitat més clara és competir amb Sisfarma amb un model de service-as-software: convertir-se en una consultoria i intentar optimitzar amb IA tots els processos.

Crec personalment que l’anàlisi i la potenciació de la venda podrien crear mercat, però requeririen ser invisibles: àudio a text, ulleres intel·ligents… I tot això implica passar informació del pacient a tercers i obtenir-ne el consentiment, especialment delicat perquè es parla de salut. Legalment impossible.

No he explorat el tooling per a grups de farmàcies. Crec que aniran creixent i que pràcticament desapareixeran les farmàcies realment independents, de manera que hi ha un mercat emergent (comparatives entre farmàcies del mateix grup, benchmarks, gestió centralitzada de marques pròpies, stocks, reposicions…).

Una altra opció és posicionar-se del costat del laboratori o la marca. Tenen pressupost per educar el farmacèutic i per interactuar-hi comercialment. Es pot intentar automatitzar aquesta feina, o fins i tot a donar eines al farmacèutic (finançades pel laboratori) per treure fricció, facilitar processos o augmentar-ne el coneixement.

I, finalment, construir un ERP modern. El que caldrà és esperar uns anys, que hi hagi relleu generacional i que passin les regulacions del col·legi de farmacèutics. Segurament vindrà per part del propi ERP.

Conclusió

A la farmàcia, s’adopta tecnologia quan resol un workflow obligatori: l’ERP per la recepta electrònica, Blistersuite per la regulació.

No veig cap producte sense consultoria que actualment pugui crear mercat en aquest espai. Crec que, en uns anys, els ERP aniran introduint aquests fluxos, però és molt difícil competir-hi.

Orkava ha mort. Tenim l’oportunitat d’entrar al mercat i substituir Sisfarma en algunes farmàcies, però des del principi vam decidir que no seriem consultors.

Temporal

Thu, 08 Oct 2026 13:07:43 GMT

Temporal

Useful (miss/un)documented articles for working with Temporal.

Addictions

Thu, 08 Oct 2026 13:07:43 GMT

Addictions

Addictions determine who you are. The fight against them shapes who you will become.

Why I avoid them

Because they take from me things I am not willing to give up:

Desire for greatness

I want to be great. I believe I can be, I just need to keep improving.

Growth lies in discomfort and small work that compounds. These require boredom and thought.
Dreams fall short under the urgency to fulfill the present moment.

Innocent belief in perfection

I resonate with Plato’s idealism. I have always been distressed by the lack of perfection. Plato’s Forms are perfect and unchanging.

Addictions have no existence of their own. You are a vessel of their existence.

Novelty

Addictions are thieves of novelty. We disregard everything to bathe in their embrace.

I’ve seen it happen with weed. A friend stops showing up. Why go out when you already know what will make you feel good?

Some not-so-obvious addictions

Porn

There is a societal consensus that tells us it’s okay.
Most of my friends talk about it with shame, that it doesn’t feel okay. On finish, there’s repulsion.

I quit porn around a year ago. I simply decided to stop. I read this book.
Since then, I’ve been at my best physically, and my happiness has increased. It might be cause or effect.

Coffee

Coffee is the most socially defended addiction.

Most daily consumers deny that they are addicted, yet openly attribute their mood, productivity, and patience to whether they have had it.

The usual defense is that it’s not bad for your health. This misses the point. Harm is not what makes an addiction worth avoiding.

Love

Being in love is the strongest addiction I have ever experienced. It robs you of everything, only that one girl matters.

If mutual, this is the only addiction I would welcome. Even knowing the cost.

Digital Consumption

Likely the addiction that most affects our generation. I used to watch Instagram every night before going to sleep, as if it were a ritual. I now am addicted to YouTube.

Everyone participates in it. It robs you of attention. A quiet moment comes, and you fill it before it begins.

Fighting addictions

My beliefs are strong. My will is weak. If something relies on my willpower, I will fail.

Avoiding temptation

The best way to fight addictions is not having to. It’s hard to not eat a cookie, it’s much easier to not buy it.

Self-honesty

Do not attempt to ‘reduce’ an addiction, to drink less coffee, to watch porn only every x days. Moderation is self-deception.

Quitting requires a decision and acknowledgement that you were an addict. Note the past tense.

Compounding effect

Addictions compound. One must always be vigilant, keeping addictions at bay. It’s easy once you fall back to one to wish for another.

I have to fight less every day.
I’m building a life where addictions don’t belong.

How I Want to Live My Life

Thu, 08 Oct 2026 13:07:43 GMT

How I Want to Live My Life

Why am I writing this

I have a burning desire to do more. To be better.
I think I have always had it, but it was silenced over the years. Now it’s roaring back.

No matter how many times I lose, I know I am going to win.

How my life has been up to now

When I was young, I got used to being good at everything.

I was talented and lacked ambition, so I didn’t put in the effort.

When I went to uni I expected it to be hard. It wasn’t.

At that point in my life I couldn’t care about anything.
I was unhappy.

Then I fell in love. I failed hard.
It made me happy.

Failure was a hard concept to grasp. Now I pursue it deliberately through physical effort.

I have not had a clear path forward. I tumbled from one thing to another until my current job.
I dedicated a year to it, every day, all day. I am proud of the work done.

But now that I know what I want, I am trapped by its success. I can either wait, or pursue my path and walk away from the reward.

Physical Effort

I used to think that achieving without effort made me better.

I was wrong. I have fallen in love with pain.
I now push myself for the effort itself.

Coding

I came to hate coding.

There has been a breakthrough since. AI.

Coding this site has been so much fun. Building fire is a joy.

Happiness

Happiness is a pursuit that dies on its existence.

I have only found it through commitment.

How I will do it

The only thing that matters is winning.
We can add artificial constraints, but those are only visible to oneself.

A goal has to be bold enough to seem unbelievable to most people yet still reachable in your own mind.

This past year I started setting goals. So far I have mostly failed.
But I will not stop.

Most people live their lives without ever winning.

I started tracking my workouts

Dream state

If you don’t choose a direction, you will end up living under someone else’s dreams.

For most of my life, I’ve been guided by what I didn’t want to do.
That’s inefficient. There are an infinite number of options to reject, but only one to choose.

My lifetime goals are:

To make a meaningful positive contribution to humanity
To live in complete freedom

It is hard to choose a path that combines both, and I am uncertain how or with which to start.

These goals are not unconditional, they are constrained by fairness.
I want to give back to those who deserve it.

I want to keep the fire burning.

Most days feel like I do something I don’t want, after doing something I didn’t want.

Many people optimize for comfort in the present.
I have my sight set at eternity.

2026

Thu, 08 Oct 2026 13:07:43 GMT

2026

This will be the third year I set goals for myself for the year to come.

I want them to be ambitious up to the point where I can only achieve them with dedication throughout the year.

Goals

Bench 110kg
Run a sub-90min half-marathon

Both goals are easy to measure and to work towards.
Additionally, I want to invest more time in building projects and relationships.

Oh, I also won’t drink alcohol.

Sport

Thu, 08 Oct 2026 13:07:43 GMT

Sport

I strive for sport to be part of my live

Past

Growing up I played tennis. I was good, competed at province level. I disliked training, but I loved competing.

When it got to either train daily or stop competing at high level; I stopped.

Since then I barely did any sport.

At 23 I started boxing. At 24 I started running consistently, boxing daily and going to the gym with a purpose.

In my first boxing fight I realized I’d lost my competitiveness. It felt bad. I was unsure if it would come back.

Present

I hate running. I hate lifting. I dislike getting punched in the face.

I enjoy pain. It makes me feel alive, brings me down to earth.

I’ve mechanized my day: Wake up -> box -> work -> run -> finish work -> gym/run -> rest. I’ve gradually built up to it, and most weeks I end up exhausted.
The hardest part is starting. It’s easy to find excuses.

I have so much curiosity to find out how far I can push my body.

It’s very rewarding to see progress. In running and lifting it’s easy to measure.

I recently did another fight, and I had my competitiveness back.

My physique is improving. It makes me feel good.

Future

I don’t know the form my schedule may have, but I will keep growing.

WebMCP

Thu, 08 Oct 2026 13:07:43 GMT

WebMCP

This site is machine-friendly: every page doubles as a WebMCP server — a proposed W3C standard that lets AI agents running in your browser discover and use tools exposed by the page you’re visiting.

WebMCP: checking…

Tools available to browser agents

search_site read — semantic + keyword search over every page of this site.
read_page read — full text of any page, by path.
ask_site read — ask a question; get a generated answer grounded in this site, with sources.
get_site_map read — list of all page paths.
contact_miquel write — send the site owner a message on the user's behalf (agents confirm with the user first).

These tools are registered on every page of this site, not just this one — this page exists to show their status and document them. They follow the WebMCP proposal (Chrome origin trial — enable chrome://flags/#enable-webmcp-testing, or the site needs an origin-trial token).

Agents without browser WebMCP can use the hosted MCP endpoint at https://api.miquelpuigturon.com/mcp.

How it works

When a WebMCP-capable agent visits any page on this site, the tools listed above are registered through the browser’s modelContext API. The agent can then search everything I’ve written, read any page, ask questions grounded in the site’s content — or send me a message on your behalf (it will ask you for your name, email and message first; that tool is marked as a write, so agents confirm before using it). All of this runs without any server-side integration, using your own browsing session.

On this site, the tools are backed by Cloudflare AI Search, which crawls and indexes all pages (synced daily).

For agents without browser WebMCP

The same search capability is exposed as a hosted Model Context Protocol endpoint on this site’s own API domain, plus a small public HTTP API:

MCP endpoint: https://api.miquelpuigturon.com/mcp
Search API: POST https://78a22e35-6a11-4b9a-84fe-1ee03a805a39.search.ai.cloudflare.com/search with {"query": "..."}
Ask API (OpenAI-compatible): POST .../chat/completions

Status

WebMCP is experimental: it is shipping as a Chrome origin trial. Until it lands, the tools activate only in browsers that enable it (e.g. via chrome://flags/#enable-webmcp-testing). The hosted MCP endpoint above works everywhere, today.

Pain, Suffering and God

Thu, 08 Oct 2026 13:07:43 GMT

Pain, Suffering and God

I always knew I wanted to build something mine.

I did not expect the amount of pain and suffering that would come with taking the leap of faith.

I now believe that pain and suffering are warranted.
And that success is a gift from God.

I see myself as a pirate searching for a treasure, sailing a violent sea. Every now and then, a new storm to sail through.

Pain

Nothing is given. Every little thing requires effort, and most effort goes unrewarded.
At first it’s annoying, then angering, at last depressing.

Pain is the only guarantee.

Suffering

Pain is unavoidable. You choose how much to suffer.

The crew is very important, shared pain leads to laughter.
When you decide it’s okay to sink and there’s someone that refuses to, you live until the next storm.
The wrong crew leads to the most suffering.

God

Finding the treasure is mostly luck. When I think about it I see God. As faceless as I am faithless.

I can curse him, thank him and everything in between, it only changes my mood, which is enough.

All in all

The world is not made for people who want to be different. They are rewarded with pain and opt into suffering beyond what they imagined.
It’s ok, brings them closer to God.

Failure is a choice, success is a gift from God. You can always choose pain.

I have had my ups-and-downs. I’ve wanted to quit: throw the boat, stay on land.
I’ve wished not to find the map so I could stop searching for the treasure.
But God had other plans, picked me up when I was at my lowest.

I want to smile to the pain, sail the seven seas, and if I have to fight to stay afloat, so be it.

OOM investigation

Thu, 08 Oct 2026 13:07:43 GMT

OOM investigation

A production debugging story: from “everything is crashing” to finding a 1-line root cause buried under 12MB of tokenizer data.

The Setup

We run a Temporal TypeScript worker that manages long-running conversation workflows. Each workflow orchestrates multi-channel interactions with leads. The worker runs on ECS with 4GB task memory (~3.8GB for the container), configured with:

workflowThreadPoolSize: 2
maxCachedWorkflows: 300
workflowTaskSlotSupplier: { numSlots: 12 }
reuseV8Context: true (default)
--max-old-space-size=3072

This configuration had been stable for months. It is explained in worker configuration.

The Problem

February 6, 11am. After a routine deployment, our conversation worker instances started dying in a loop. ECS kept restarting them, they’d run for about 60 seconds, then get OOM-killed. Every single instance, every single time.

The deployment included ~25 commits: an AI SDK v5-to-v6 migration, a URL parsing refactor, a new “previously contacted” substatus, task management changes, and various smaller fixes. Nothing that screamed “memory bomb.”

Phase 1: Rule Out the Obvious

Hypothesis: Outlier Campaign Graph Loop

This was the first and longest part of the investigation, because we’d been bitten by this exact pattern before.

Each campaign in our system is modeled as a directed graph - nodes are sequence steps (send email, wait, check reply, branch on condition, etc.) and edges define the flow between them. A conversation workflow walks this graph as leads progress through the campaign. If the graph contains a cycle - a loop back to an earlier node - building the graph can cause infinite recursion. We’d had a previous incident where exactly this happened: a malformed campaign graph caused workflows to loop, their in-memory state ballooned, and the workers OOM’d.

So when the same symptoms appeared, this was the natural suspect.

Tests:

Queried all campaigns active around the incident window (~11am Feb 6), looking for recently created or modified campaigns that could have introduced a loop.
Inspected campaign graphs for cycles, focusing on campaigns with high conversation counts.
Measured individual conversation workflow state and sequence payload sizes to look for the telltale ballooning pattern (a looping workflow accumulates megabytes of step history).
Tested specific suspected campaign IDs by isolating them - checking whether stopping certain campaigns would stabilize the workers.
Filtered to active-only conversation subsets to narrow the search.
Cross-referenced campaign launchedAt / updatedAt timestamps with the exact incident start time.

Result: No outlier. Conversation state sizes were normal (not the multi-MB explosion you’d see from a loop). No campaign graph had an obvious cycle. No single campaign’s removal stabilized things. The pattern didn’t fit: a graph loop causes one or a few workflows to explode, but here every instance was dying regardless of which workflows they happened to cache. This pointed away from a campaign-specific issue toward something systemic.

Still, this phase consumed significant time because the previous loop incident had been so similar in symptoms. Too much time was spent here.

Hypothesis: Retry Storm / Stuck Activities

Maybe a subset of workflows or activities were retrying aggressively and driving memory + CPU.

Tests:

Temporal CLI checks on running workflows and pending activity attempts.
Focus on conversationWorkflow and conversationCallManagerWorkflow.
Added interceptor logging at activity start to capture which activities were firing, their workflowId, and retry attempt number - this let us see if any activity was retrying excessively across the fleet.
Added workflow replay logging to detect if workflows were being replayed repeatedly (which would indicate nondeterminism errors or sticky cache eviction churn).

Result: Some noisy workflows existed, but no single smoking gun retry loop explaining cluster-wide OOM. A known nondeterminism error existed in one workflow, but a single errored workflow doesn’t explain fleet-wide memory growth. The interceptor logs showed normal activity distribution - no hot retriers dominating the fleet. Ruled out.

Hypothesis: Nondeterminism Error

The deployment included a refactor of getPublicIdentifierFromUrl that changed URL parsing behavior. In Temporal, changing the behavior of functions used in workflow code breaks replay determinism. We found 46 conversations with protocol-less URLs that would hit this.

Test: Reverted the change and deployed. Result: Still crashing. Ruled out.

Hypothesis: AI SDK v6 Memory Increase

The AI SDK upgrade was the biggest change. We measured heap usage by importing the old vs new packages in isolation.

Test: Compared process.memoryUsage().heapUsed with AI SDK v5 vs v6. Result: +5.3MB heap difference. Not enough to explain multi-GB growth. Ruled out as sole cause.

Hypothesis: Node.js Version Change

Our Dockerfile used node:22-bookworm-slim (a moving tag). It had silently updated from 22.21.1 to 22.22.0 on Feb 3.

Test: Pinned to FROM node:22.21.1-bookworm-slim and deployed. Result: Still crashing. Ruled out.

Phase 2: Observing the Crash Pattern

At this point, every obvious hypothesis had been eliminated. We stepped back and looked at the external monitoring data.

Sticky Cache Sawtooth

The temporal_sticky_cache_size metric showed a repeating sawtooth pattern. Workers would start, the sticky cache would fill as workflows got assigned, and then the instance would die. Restart, fill, die. Over and over.

ECS Memory Utilization

ECS MemoryUtilization told the same story from the infrastructure side. Both average (blue) and maximum (orange) climbed, then dropped sharply as containers were OOM-killed and restarted.

The correlation was clear: memory grew in lockstep with the number of cached workflows. Each workflow the cache absorbed pushed the container closer to its limit. This shifted our thinking from “some bad workflow is exploding” to “every cached workflow costs more memory than it should.”

Phase 3: Adding Diagnostics

We were still flying blind on why per-workflow cost had changed. process.memoryUsage() only tells you about the main thread’s V8 heap. We needed visibility into RSS (total process memory) to see what’s happening outside the heap.

We added a 10-second interval logger:

setInterval(() => {
  const mem = process.memoryUsage()
  const heapUsedMb = Math.round(mem.heapUsed / 1024 / 1024)
  const rssMb = Math.round(mem.rss / 1024 / 1024)
  const externalMb = Math.round(mem.external / 1024 / 1024)
  const arrayBuffersMb = Math.round(mem.arrayBuffers / 1024 / 1024)
  const estimatedNativeMb = Math.max(0, rssMb - heapUsedMb - externalMb - arrayBuffersMb)

  console.log('Worker diagnostics', {
    workerState: getState?.(),
    workerStatus: getStatus?.(),
    heapUsedMB: heapUsedMb,
    rssMB: rssMb,
    estimatedNativeMB: estimatedNativeMb,
    rssDeltaMB: rssMb - previousRssMb,
    heapUsedDeltaMB: heapUsedMb - previousHeapUsedMb,
  })
}, 10_000)

The workerStatus was key - it gives numCachedWorkflows, which let us correlate memory with cache size numerically, not just visually.

First Diagnostic Results (12 workflow task slots)

Cached WFs	Heap (MB)	RSS (MB)	Native (MB)	Delta/10s (MB)
~20	280	1,389	~1,100	—
~80	300	2,950	~2,650	+780
~100	314	3,697	~3,383	+374
killed

Two immediate observations:

Heap was fine. 280-314 MB, stable. The V8 garbage collector was doing its job.
RSS was exploding. Growing by 400-800 MB every 10 seconds. All outside the heap.

The gap between RSS and heap (what we called “native” memory) was the problem. This includes the Temporal Rust Core bridge, V8 worker thread heaps, and compiled code. But 3.3 GB of “native” memory was absurd.

Phase 4: Reducing Configuration

Hypothesis: Too Many Concurrent Workflow Tasks

Maybe 12 concurrent workflow slots created too many V8 isolates at once.

Test: Reduced workflowTaskSlotSupplier.numSlots from 12 to 6. Result:

Cached WFs	Heap (MB)	RSS (MB)	Native (MB)	Delta/10s (MB)
61	247	1,597	1,320	+434
111	230	2,176	1,922	+579
124	243	3,200	2,931	+1,024
killed

Still crashing. Slower, but same pattern. The cache fills to 124 in ~40 seconds, RSS hits 3.2 GB, OOM.
In hindsight this was obviously not the issue since V8 isolates are reused.

What the Data Told Us

From snapshots 1 to 2: 50 more cached workflows cost +602 MB of native memory. That’s ~12 MB per cached workflow.

With reuseV8Context: true, the Temporal SDK shares a single V8 context across all workflows per thread. Per-workflow memory should be ~1 MB (just the workflow’s local state). 12 MB was 12x too high.

Cross-Reference: The Enrich Service

Our other Temporal worker (enrich service) uses the same @enginy/common package, the same AI SDK v6, and has maxCachedWorkflows: 350 with less container memory (3.5 GB). It wasn’t crashing.

This told us the problem was specific to what’s inside the conversation workflow bundle, not the shared infrastructure.

Phase 5: Finding the Root Cause

With the per-workflow cost established at ~12 MB and the enrich service working fine, we turned to the workflow bundle itself.

Temporal workflow code doesn’t run directly from your source files. Instead, the Temporal TypeScript SDK uses webpack to produce a self-contained workflow-bundle.js (or you can generate it at build time like we do). This bundle is what gets loaded into V8 worker thread isolates.

The reason for this separate bundling step is determinism: the SDK needs to control the execution environment inside the isolate, patching non-deterministic APIs like Date() and Math.random() with deterministic replacements so that workflow replay produces identical results. Activities run on the main thread and can import anything normally, but workflow code can only access what’s in the bundle.

This means whatever ends up in the bundle gets loaded into every V8 worker context. And anything that shouldn’t be there - that wasn’t there before - directly inflates per-workflow memory cost.

What was in the conversation bundle that wasn’t in the enrich bundle?

We grepped the built workflow-bundle.js:

grep -n 'gpt-tokenizer\|creditsCostCalculator\|encodeChat' conversation/dist/workflow-bundle.js

The result was damning. The entire gpt-tokenizer package - including its ~2 MB BPE rank tables with 200,000 tokenizer merge rules - was embedded in the workflow bundle. This data gets decoded into ~20 MB+ of in-memory JavaScript objects when the V8 context evaluates the bundle.

Before the fix:

Bundle size: ~12 MB
Heavy-module hits in bundle: 96 (ai-sdk / tokenizer related)

The Import Chain

The workflow code legitimately imports utilities from @enginy/common/browser:

// conversation/src/workflows/utils/conversationIdentityActionsReceiver.ts
import { getMillisecondsToNextWorkingHour, isInWorkingHours } from '@enginy/common/browser'

browser.ts is a barrel file with ~130 exports. A recent commit (8ecd24ed9, Feb 5 - “AI Variable Credits Not Showing Full Cycle Cost”) had added one line:

export { getPromptCreditCost } from './credits/creditsCostCalculator'

creditsCostCalculator.ts imports encodeChat from gpt-tokenizer. No workflow code uses getPromptCreditCost. Webpack should have tree-shaken it out.

Why Tree-Shaking Failed

grep 'sideEffects' common/package.json
# (empty)

@enginy/common’s package.json had no sideEffects field. Without "sideEffects": false, webpack must assume that importing any export from a module might trigger side effects. It can’t safely remove getPromptCreditCost because the act of importing creditsCostCalculator.ts might do something important (register a global, modify a prototype, etc.).

So webpack included creditsCostCalculator.ts, which pulled in gpt-tokenizer, which pulled in its BPE rank tables, encoding params, model maps, and everything else.

The Fix

Added one field to common/package.json:

{
  "sideEffects": false
}

This tells webpack: “every module in this package is pure - if you don’t use an export, you can safely remove it.” Webpack then tree-shakes getPromptCreditCost and its entire dependency chain (gpt-tokenizer) out of the workflow bundle.

After rebuild:

Bundle size: ~2.9 MB
Heavy-module hits: 0

After the Fix (Runtime)

Cached WFs	Heap (MB)	RSS (MB)	Native (MB)	Delta/10s (MB)
0	195	501	285	—
4	192	535	322	+34
29	199	543	322	+5
78	214	638	386	+2
92	189	593	376	+1

92 cached workflows at 593 MB RSS. Before the fix, 61 cached workflows was already 1,597 MB. Per-workflow cost dropped from ~12 MB back to ~1 MB.

The base RSS dropped from ~1,200 MB to ~500 MB - the gpt-tokenizer tables were consuming ~700 MB just being loaded into the two V8 worker thread contexts.

Lessons

sideEffects: false matters. If you maintain a shared library consumed by webpack, declare it. Without it, a single barrel export can drag megabytes of unrelated code into your bundle. In our case, a tokenizer that no workflow ever calls.
Barrel exports are a liability in bundled contexts. Our browser.ts has 130+ exports. Any one of them can pull in unexpected transitive dependencies. For performance-critical bundles like Temporal workflow bundles, consider importing from specific subpaths instead of barrels.
process.memoryUsage().heapUsed only covers the main thread. In Temporal workers, activities run on the main thread while workflows run in separate V8 worker threads. If heap is stable but RSS climbs, that tells you the problem lives in the workflow threads (bundle, V8 worker contexts), not in activity code. It narrows where to look, but the actual root cause still requires inspecting the bundle and correlating memory with workload counters.
The Temporal workerStatus object is invaluable. numCachedWorkflows let us correlate memory growth with cache size, which pointed us toward per-workflow cost as the problem.
Inspect built artifacts, not only source. The bug was invisible in the source code - no workflow imports gpt-tokenizer. Only by grepping the built workflow-bundle.js did we find the 12 MB of dead weight. When memory regressions appear after a deploy, validate what actually lands in the bundle.
The investigation loop works. Every phase followed the same pattern: form a hypothesis based on available data, design a minimal test, deploy, observe. Most hypotheses were wrong. That’s fine - each one narrowed the search space.

Worker configuration (Node)

Thu, 08 Oct 2026 13:07:43 GMT

Worker configuration (Node)

This is a walkthrough of my experience configuring workers running on Node for Genesy.

It focuses on worker configuration, as I found the current (as of Oct 2025) defaults to be misguiding, and want to share my findings, as this is not only a me-issue.

Readers should understand workers and worker/server interaction before reading this article. (a good resource)

Fun fact: This article is the reason this site exists.

Background

Click to expand

We run two services Each service simply runs a worker

Enrich: User actions are not processed in the API directly, they are off-loaded to this service, which makes sure that they are processed reliably. I wrote my CS bachelor’s thesis on why we decided on this, and how I coded the primitives that allow it to work well.
Conversation: Genesy offers automated campaigns, which are likely the most advanced conditional campaign system that exist on any tool. Each campaign has an identity which targets contacts, with which a conversation is started. Each conversation runs as an entity workflow, and identity actions are centralized in a manager workflow.

Even though the learnings apply to both services, most of the investigation comes from the Conversation service, as we want it to scale to infinity reliably (started in February with <50k active conversations, in October it handles >500k).

Why did I start looking into this

Users can see conversations in Genesy via the inbox. The inbox also allows sending messages directly.

There’s nothing worse for users than sending a message twice. To avoid this, we use updates to prompt the manager workflow to send the message, and get a response (whether it was successful or not).

If the manager workflow is not responsive, then the update fails, and the user is sad. As my main job is to make users happy, this had to be solved.

Priority

6th of march, Temporal announced priority and fairness.

It took 3 months of me complaining that it wasn’t working, until priority was an actual useful feature.

I thought that priority would solve this problem. Simply prioritize manager workflows over conversation workflows. Then we only need to make sure that we can handle hundreds of workflows immediately instead of hundreds of thousands.

Discovering the problem

I started experimenting with how to make sure priorityKey=1 workflows are always responsive no matter how many higher priority workflows needed processing. An easy way to stress test it was to limit the number of instances of the service, and thus of workers.

Previously, up to 50 instances were running at a time, as our best guess on making priorityKey=1 workflows responsive was to process everything instantly.
I cut down to 10 instances, and at peak times, workers were saturated By saturated I mean that for the task queue, the backlog increases: tasksAddRate > tasksDispatchRate .

Stats for the conversation task queue by priority key

Even though it looks great, as the backlog age is less than one second for priorityKey=1, it performed worse.
That day, we went from ~97% success on users sending messages to ~85%.

At this point, tasks were picked up by workers almost instantly, yet they failed. Why??
Looking at metrics, many workflow tasks were failing due to status_code=not_found

Workflow task failures and latency

It’s obvious from the image that the failures are related to the task execution latency Time since the worker picks up the task until it responds to the temporal server . kappa agreed:

metric temporal_request_failure with operation=RespondWorkflowTaskCompleted and
status_code=NOT_FOUND indicates that a worker attempted to report the completion
of a workflow task, but the Temporal service could not find the corresponding
workflow task. This typically happens when the workflow task has already timed
out—meaning the worker did not respond within the default 10-second timeout window

The worker was taking more than 10 seconds (default start-to-close timeout) to complete the task.
This was a surprise, as workflow tasks are lightweight, and should take tens of milliseconds at most.

Understanding worker configuration

The worker configuration is defined on the worker.create method. Here we focus on performance-related options.

Task execution and polling

Each worker has a number of pollers and slots for each task type workflow/activity/local-activity/nexus . The pollers grab tasks from the Temporal server, and occupy a slot while the task is being executed. The reasoning behind having multiple pollers and not just one is due to performance. From the code docs:

In general, a Workflow Worker's performance is mostly network bound (due to
communication latency with the Temporal server). Accepting multiple Workflow
Tasks concurrently helps compensate for network latency, until the point
where the Worker gets CPU bound.

There are two ways to configure this:
The original way, using maxConcurrent <taskType> workflow/activity/local-activity/nexus task Executions and maxConcurrent <taskType> workflow/activity/local-activity/nexus task Polls.
The new way, using the tuner, which grants slots until a targetCpu and targetMemory are reached.

We will focus on workflow and activity tasks, which are the majority of tasks that are executed.

Threading model

The threading model is explained here.
Basically, interactions with the Core SDK happen on the main thread (Node’s event loop). There’s also at least one worker thread, which the main thread delegates workflow tasks to. This is a great idea, as workflow tasks are CPU bound (they run until the next await).

The number of worker threads is configured with the workflowThreadPoolSize option. It defaults to 1 if reuseV8Context is true, or 2 otherwise.

Workflow tasks tend to be lightweight, so the defaults are usually fine.

Sticky tasks and worker cache

There’s also the nonStickyToStickyPollRatio, which indicates the ratio of pollers that will poll from the sticky task queue.
Along with maxCachedWorkflows, it is a very useful optimization to avoid having to replay workflows.

The values should be set according to your workload patterns.
The docs on it are good, so no need to go deeper here.

What can we do to fix the problem?

Task execution and polling

By far the most relevant configuration is related to task execution and polling.

Looking at metrics for number of slots slots are tasks being executed concurrently , we can see that the number of workflow slots is directly related to the number of tasks failing, and through the previous relationship, the number of workflow tasks timing out.

Workflow task failures and slots

This image is obtained using the tuner, targeting 0.8 CPU and 0.6 memory usage, with a workflow task rampThrottle time before handling new slots of 100ms. Without the rampThrottle, we would get a lot of “out of memory” errors, as the worker would fill its memory trying to execute too many tasks at once.

It’s obvious from the image that the tuner is granting too many slots, and that this for some reason, is causing tasks to fail.

The worker takes more than the start-to-close timeout to complete the tasks, so when responding to the Temporal server with a result, the task is no longer found, and the result is discarded. This is very bad, as that same task is reissued, and the process repeats, burning CPU cycles for no reason.

Key Insight: Workflow tasks take so long to complete, not because they are expensive (they take tens of milliseconds), but because there are many slots executing concurrently along with the main thread.

Threading model

In all my tests, the tuner grants way too many workflow slots no matter what targetCPU is specified. To understand why, we need to look at the threading model.

CPU usage for targetCPU=0.5 (50+ slots)

This is running on ECS Fargate, on single vCPU instances.

Running 2 vCPU instances, with the default configuration, the tuner was slightly more useful, but still granted too many slots.

2 vCPUs slots and task failures

When configuring a fixed sized workflow task slot supplier, with 2 vCPUs and one worker thread, I could get a bit less than twice the number of slots as with 1 vCPU, but maximum CPU usage was less than 75%, which makes sense, the worker thread was at full capacity, and the main thread was half-idle, mostly due to I/O bound work.

2 vCPUs CPU usage

Conclusions

I couldn’t get the tuner to work as expected, so I can’t recommend using it, it causes too many workflow task slots, which makes them time out and be retried, burning CPU cycles for no reason.

The most important thing is to use a fixed sized workflow task slot supplier, with a size that is a good fit for your workload.

Workflow task failures and slots

If no configuration is used (no tuner, no maxConcurrentWorkflowTaskExecutions), the worker will use the default configuration, which is 40 workflow slots (likely still way too many).

Why is the default worker configuration so bad?

Well, It’s not that bad…
For parallel programming languages (like go and java, which are likely Temporal’s target languages), it makes sense, as we want to reduce the percentage of I/O work, and there’s no event loop or main thread that limits performance.

I have explored more in detail the tuner, running on 2vCPU instances, and poller configuration, which you can read about in the deep-dive article.

A decent worker configuration

~~We now use~~ A good enough configuration is the following:

const config = {
    ... other things
    tuner = {
        workflowTaskSlotSupplier: {
            type: 'fixed-size',
            numSlots: X (5 for conversation, 10 for enrich),
        },
        activityTaskSlotSupplier: {
            type: 'resource-based',
            rampThrottle: '100 ms',
            tunerOptions: {
                targetCpuUsage: 1.0,
                targetMemoryUsage: 0.6,
            },
        },
        // local and nexus tasks don't really matter
    },
    nonStickyToStickyPollRatio: Y (2/5 for conversation, 0.5 for enrich),
    maxCachedWorkflows: Z (150 for conversation, 300 for enrich),
}

The difference between conversation and enrich is that conversation workflow tasks are slower, due to replay, and more of the conversation workflows end up not being sticky.

Values for activity tasks are not very relevant, as our workflows don’t fan out into many activities, and eager dispatch is enabled.
Activities are mostly processed instantly, and are very cheap, though they do sit in the event loop, adding a bit of latency to the main thread.

Workflow tasks latency and failures with 6 slots

Running the same load, we can see that with 6 slots, there are no failures, and that the maximum time that workflow tasks take to process is close to the start-to-close timeout. With this configuration, 100% of users sending messages succeed.

Why not just 1 slot?

Even though the default is not good, the docs are correct in that worker performance is mostly network bound so it’s still useful to have multiple slots. We want to make sure that either the workflow thread or the main thread are never idle due to having no work to do.

Running workers in production

This article is related to worker configuration, but to run workers in production, infrastructure and scaling strategies also need to be considered.

Workers should run on fixed-size instances and scale horizontally. (We use ECS Fargate, and run 1-50 (usually under 10) instances per service, larger teams probably use Kubernetes)

Scaling should be done on average CPU usage across worker instances, and either task backlog count or backlog age. (We send task queue metrics to CloudWatch)

Boxing

Thu, 08 Oct 2026 13:07:43 GMT

Boxing

The first part was originally hand-written in Catalan

Why Do I Box?

It’s a question I often ask myself.

I don’t do it for the reasons one would imagine:

I don’t care about the ‘aura’ it gives me
It doesn’t help me relax
Even though physical superiority makes me feel good, it doesn’t compensate for the bitter aftertaste of being the one who inflicts pain

This is what is most clear to me. Makes it hard to justify following through.

Every day, I wake up and go back to it.
Boxing forces me to face myself: am I enough, will I still show up knowing I’ll get hit, and do I have the competitive drive to keep improving.

Prologue

The boxing club is in the building across from where I live.
I used to look at it with curiosity.

Walking with a friend, the topic came up. I said I’d like to try it out.
My main reason was the nobility of the sport. Also, I thought knowing how to fight was worth something.

My friend said she couldn’t imagine me doing it. That I was incapable of hurting someone, and I’d lack the malice both to perform and to have the will to do it.

Beginning

Boxing has more technique than I expected. The first half of the year I didn’t fight. I remember when I asked for it, the first sparrings were light. People had good faith, and they wanted to help.

At this moment, I thought I was a decent fighter. I saw what I could improve on, and I was ready to work on it.
I was ecstatic about the decision of boxing, which so far confirmed the nobility of the sport.

Reality Check

The coach left, and a guy who trained with us, became the new coach.
He is one of the people I like most from everyone I’ve met. We’ve barely spoken outside of boxing, I know nothing about him.

The first painful sparring I remember was with a fat guy, who was as inexperienced as me, but didn’t have the blocker I have of not wanting to cause harm. I went a few days with a headache, and that was it.

Then an interclub came up, and with my coach’s encouragement I signed up. I approached it with a mix of desire and confidence, with the mentality of thanking the opponent to fight with me, to do a slightly harder sparring.
He went out to kill. I got hit, and I struggled to defend myself to reduce the damage.

That day, I wrote:

The fight was disappointing. I wasn’t beaten up, but I didn’t fight either. I could look for a thousand excuses, but there’s no need.
Noted for the next one to do better.

End?

Since the interclub, I’ve been boxing every day.

I had my nose broken, I’ve bled more times than I can count on one hand.
The worst moment was when I made the other bleed. Noticing it made me not throw any punches. I felt awful.

Now another interclub is coming. My trainer recommended I sign up. I did it.

To train for it, my coach invited me to a neighbour club to spar. It was tuesday.
That day I woke up, trained boxing, and the afternoon I went to the club. I got beaten up. They were better, and as a souvenir, they gave me pain dripping with blood.
I got home feeling like crying, what hurt most was the realization of my inferiority. I called my mom, couldn’t tell her as I didn’t want her to worry.
That night the pillow soothed my crying.

I have less desire and I am less confident, but I’m also more prepared than the first time.

I wrote everything until here the night before the interclub

Post Interclub

I crushed him. I was superior. If it were Sant Jordi, I would have made a rose out of his bleeding.

I am confused. On one hand, I am proud. On the other hand, I feel awful.
Between rounds I told my trainer that I didn’t want to hurt him. The fight was stopped a couple of times to wipe the blood off his nose. I didn’t hit him as much as I could.

It doesn’t make sense to continue boxing. The nobility is permanently stained of violence much like his white shirt with blood.

I don’t enjoy victories. I assume them. Maybe that’s why losses bother me so much.

Since I stopped competing at tennis, there was something uneasy within me. I wasn’t competitive, I accepted defeat.

Today I wanted to win.

I have shut down the inquietude, but I have sparked a fire inside me.

I want to compete.

2 weeks later

Decision

Today I announced I was leaving the club. It was shocking to my trainer, but it felt right.

The main driver behind the decision is that I want to compete. Next step would be an amateur fight, but I don’t want to do it. I am not scared, but in boxing, higher level means more damage. I don’t want to inflict damage.

Also, I have ambitious goals in running and lifting, and going to box every day is not helping me reach them. Especially lifting heavy, as the tension of the fight fries my CNS.

Finally, I have always said that when I move out of my current flat, I would stop boxing, and I’ll likely leave in the next months.
I’m just getting ahead and making it my choice instead of waiting for life to take care of it.

Boxing has been a great reminder of who I am.
Ambitious, capable, and decisively naïve about how the world actually works.

OOM investigation > Enrich deep dive

Thu, 08 Oct 2026 13:07:43 GMT

OOM investigation > Enrich deep dive

A sequel to the conversation worker OOM: same class of problem, no single-line fix.

After fixing the conversation worker, I expected the enrich worker to be boring. It wasn’t crashing as dramatically, but it had the same smell: memory climbed with cached workflows and the bundle was much larger than it should have been.

The difference was that this time the problem didn’t collapse into "sideEffects": false. Instead, it turned into a bundle experiment log.

I started a thread in the Temporal Slack while working through this.

Why We Started Looking

The previous incident taught us something important: for Temporal TypeScript workers, the workflow bundle is not just a build artifact. It is part of the runtime memory model.
So after that fix, I looked at our other Temporal service, which acts as a background jobs runner for many types of workflows: enrich.

The numbers were suspicious:

Service	Bundle size	Cached WFs
conversation	~2.93 MiB	~700
enrich	~14.1 MiB	~150

That immediately put the bundle under suspicion.

The top contributors in the enrich bundle were:

zod@4 at ~1 MB
ai@6 at ~435 KB
workflow tool definitions at ~350 KB
@enginy/common at ~300 KB
workflow utils at ~227 KB
tldts at ~150 KB
a huge hardcoded isPersonalEmail domain set at ~126 KB
Prisma browser runtime
HTML parsing dependencies

The First Hypothesis

The obvious theory: each cached workflow gets its own copy of too much code, so bundle size is directly drives RSS.

That fit the observed failure shape. Workers would start, the sticky cache would fill, and memory would rise with it.

Bundle Inspection

The first pass was static analysis. I looked at the built workflow bundle and the import graph, then traced the biggest paths back to source.

syncQueue pulled in far too much

syncQueue.ts imported multiple CRM adapters directly. Those adapters brought in tldts, HTML parsing packages, string similarity helpers, isPersonalEmail, and the Prisma browser runtime.

The workflow wasn’t just orchestrating activities anymore. It was pulling a large chunk of adapter logic into workflow code.

AI workflows pulled ai + zod into the bundle

Some workflows intentionally use the AI SDK in-workflow. That is a real requirement, but it means the bundle includes ai, zod, lots of tool definitions, and large prompt/schema-related code. Legitimate, but expensive.

Can We Fix It Without Touching Workflow Code?

Before refactoring any workflow logic, I wanted to see how much we could get just by changing how the bundle is produced.

Aggressive Webpack Optimization

The first temptation: just optimize harder. minimize: true, custom Terser config, changing source map mode, more aggressive webpack optimization flags, externalizing modules.

This is where Temporal becomes opinionated.

Temporal workflow bundling is not a generic webpack build. It is part of the workflow sandbox. If you change too much, you can break workflow type names, stack traces, source maps, replay/debugging expectations, and possibly bundler assumptions used by the reusable VM executor.

In particular, minimize: true is unsafe unless function names are preserved, and replacing Temporal’s source map configuration is risky.
This is due to how workflows are resolved by name.

Result: Promising in theory, not safe enough as a default production fix. I did not want to solve an OOM by introducing a determinism or workflow registration problem.

ignoreModules / Externalization

If some heavy packages should only exist at activity runtime, maybe we can tell the bundler to ignore them.

This works only if the workflow bundle does not actually need those modules at evaluation time. That turned out to be the key limitation: if a workflow directly imports code that imports tldts, you cannot simply ignore tldts. If syncQueue directly imports CRM adapter code, those adapter dependencies are part of the workflow module graph. If AI workflows use ai and zod inside workflow code, those packages cannot be externalized away.

Result: Useful only for modules that should not have been in workflow code in the first place. Not a general solution.

Strip sourcesContent From Inline Source Maps

Temporal wants inline source maps, and changing source map behavior broadly is risky. But stripping sourcesContent while keeping line/column mappings intact is a narrower change: you still get pointers to the original file/line, but the full embedded source text is removed from the bundle.

This reduced ~150 MB of base memory and I haven’t noticed any difference in practice - sourcemap pointers still work, so errors show the same file and line information.

A smaller win compared to what could be achieved, but it was the cleanest no-refactor change.

Result: ~150 MB base memory reduction with no visible downsides.

Understanding What Temporal Actually Reuses

This was the eureka moment.

I patched the Temporal runtime locally to log what landed in the shared module cache vs the workflow-specific module cache.

The behavior is roughly:

The workflow bundle wrapper is loaded once in the reusable VM
Temporal SDK internals and a small set of known-safe modules are reused
User workflow modules are evaluated into module objects per workflow cache entry
External libraries imported by user workflow code are generally treated as workflow-specific as well

What gets reused is not “all imported libraries.” It’s mainly the shared VM context, the bundle scaffold, and Temporal runtime internals.
Reusing those blindly could be problematic. If a module has mutable top-level state: a counter, a cache, a configuration object… sharing it across workflows would mean one workflow’s execution could silently affect another’s. The SDK can’t know which modules are safe to share, so it defaults to giving each cached workflow its own copy of everything.

That explained everything. The bundle exists once, but most of its modules do not. A small workflow that only needs to verify an email was getting its own in-memory copy of the AI SDK, zod, all the CRM adapter code, the isPersonalEmail domain set, and everything else in the bundle - just because the workflow barrel imported everything eagerly.

Lazy Workflow Loading

Once that was clear, the goal became: stop every workflow from eagerly importing every other workflow’s dependency tree.

First attempt: rewriting the bundle

The first idea was to rewrite the workflow entry file to only export the workflows each type actually needs. But mapping workflow type names back to their source files is error prone, and rewriting the bundle seems too hacky, so I decided against it.

Second attempt: lazy imports

Next I tried dynamic import() to lazy-load heavy workflows on demand. This didn’t work well either: dynamic imports caused webpack to split the bundle, which breaks Temporal’s expectation of a single self-contained bundle file.

What worked: require()

The solution was using require() instead of import(). Since the bundle is already a single file evaluated in a V8 context, require() resolves within the bundle without triggering code splitting.

This meant the workflow entry file had to be a CommonJS file (.cjs), but that was a small price. The entry file maps workflow type names to require() calls, so each workflow only loads its own dependency tree when it’s actually needed.

I decided to try this with the workflows that are most run, as reducing their memory footprint means reducing it for most of the cached workflows.

The best candidates for lazy loading were workflows with large and distinct dependency trees: syncQueue, AI fill workflows and workflow runners that pull many other workflows transitively.

const lazyWorkflows = {
  syncQueue: [() => require('./workflows/syncQueue'), 'syncQueue'],
  fillAiFieldsWorkflow: [() => require('./workflows/fillAiFieldsWorkflowV2'), 'fillAiFieldsWorkflow'],
  ...
}

const getWorkflowExport = (workflowType) => {
  if (workflowType === 'default' || workflowType === '__esModule') {
    return undefined
  }

  const lazyWorkflow = lazyWorkflows[workflowType]
  if (lazyWorkflow) {
    const [loadModuleFn, exportName] = lazyWorkflow
    const workflowModule = loadModuleFn()
    return workflowModule[exportName]
  }

  return require('./workflows/index')[workflowType]
}

module.exports = new Proxy(
  {},
  {
    get(_target, prop) {
      return typeof prop === 'string' ? getWorkflowExport(prop) : undefined
    },
  },
)

When I inspected module-cache counts for specific workflows after the change, targeted workflows loaded far fewer modules than the fallback path.

Workflow	Workflow-specific modules
syncQueue	~27
verifyLeadEmailWorkflow	~19
enrichLeadWorkflow	~22
Fallback (eager path)	~280

Lazy workflow registration doesn’t make workflow code shared, but it drastically reduces how much code each workflow pulls into its module cache.
That was the real win.

Next steps will be to lazy load all workflows, though I prefer keeping it simple so other developers don’t have to understand all of this, and can keep using the simple: export * from myWorkflow

The Result

Max memory utilization

Memory utilization max dropped from ~100% with 150 cached workflows (can be seen at ~65% with ~100) to ~35% after the deployment, also with 150 cached workflows.

Max cached workflows

We ended up with a layered fix:

Strip sourcesContent from the enrich workflow bundle’s inline source maps (not a major win, but a safe starting point)
Introduce lazy workflow loading via a .cjs entry file using require() for the most common workflows
Trim barrel exports so the fallback path is less expensive

These changes improved behavior without updating workflow code, so transparent to other developers.

What Temporal can improve

Two things would make this class of problem much easier to solve:

User module opt-in freezing. If a module is known to be stateless (e.g. zod, ai, utility libraries), the SDK could allow users to mark it as safe to reuse across cached workflows instead of evaluating it into each workflow’s module cache. An opt-in freeze list would let the shared VM context actually share the expensive modules, which is what most users expect reuseV8Context to do.

Better support for per-workflow bundle splitting. The lazy require() approach works, but it’s a workaround. First-class support for registering workflows with separate bundles (or bundle segments) would let teams isolate heavy dependency trees without the .cjs entry file hack. The underlying mechanism already seems to work; making it an officially supported pattern would make this much easier to reason about.

incidentd - Architecture

Thu, 08 Oct 2026 13:07:43 GMT

incidentd - Architecture

incidentd was built around a simple idea:
strong atomic guarantees + durable eventual consistency, independent of where inputs come from or where side-effects go.

It must be incident-free.

incidentd is a standalone service that manages an incident end-to-end: from declaration to resolution. It owns state, notifications, AI suggestions, status page updates, and post-incident analysis.

From day one, it was designed to feed context into AI agents. That means data lives close, and reads/writes are fast.

Platform

incidentd runs on Cloudflare Workers, using 3 primitives:

Durable Objects -> per-incident transactional state
D1 -> queryable indexing and listing
Workflows -> async side effects and AI jobs

Working on Cloudflare has been surprisingly good: great DX, trivial deployment and low cost.

The only real downside is ISPs blocking thanks to a spectacularly bad ruling that gave power to LaLiga to control the internet. (irrelevant in this case since this is server-to-server)

The Durable Object

Each incident maps to a single Durable Object.

This is the source of truth for:

Event timeline (event_log)
Derived state: status, severity, assignee, metadata

Every mutation follows the same pattern:

Start transaction
Update state
Append event (published_at = NULL)
Schedule alarm
Commit

This is the outbox pattern.

The core idea behind incidentd is:

A single-threaded, transactional system that gives you atomicity and reliable async delivery, without distributed coordination.

Alarm-based delivery

The DO alarm drains unpublished events:

for each event where published_at IS NULL and attempts < 3:
    dispatch to IncidentWorkflow
    set published_at = now

If dispatch fails, it retries. After 3 attempts, the event is marked failed but acknowledged.
This gives at-least-once delivery with bounded retries.

The same alarm also drives agent scheduling. A single method decideAlarmAction() decides what to do next:

dispatch events
trigger an agent turn
schedule cleanup
or do nothing

This keeps the reliability story local: one place decides what must happen next, and one durable log says what already happened.

The dispatcher

IncidentWorkflow bridges the incident Durable Object and the senders. It’s a long-lived workflow that processes events in order:

dispatch first event
while incident is not resolved or declined:
    wait for next event
    dispatch it

Each event is fanned out to all senders using Promise.allSettled.

A failing sender doesn’t block others
Failures are logged, not fatal

Because delivery is at-least-once, senders should be idempotent (incident_id + event_id).

Events are committed sequentially in the DO, and dispatched in the same order.
No races, no inconsistent external state.

Why not Queues?

Queues have a classic problem: dual-writes. Update state in one system, enqueue in another. If enqueue fails, the event is lost.

It can be fixed (outbox tables, polling, distributed transactions), but complexity explodes.

Here, the outbox lives inside the Durable Object, and dispatch goes directly to a workflow.

Also:

Workflows are basically queues with better ergonomics

So I use them.

IDs and identifiers

Internal IDs: newUniqueId() → stored in D1
DO resolution: idFromString
External IDs: Slack, dashboard, etc. → stored as routing metadata

Originally I used idFromName (e.g. Slack thread ID), but that made IDs non-reusable after resolution.

The plugin model

The system is structured as: receivers → handler → durable object → dispatcher → senders

Each stage is decoupled

Receivers -> validate + normalize input
The handler -> pushes events to the DO
DO -> commits state + events atomically and schedules dispatch
The dispatcher -> delivers events
Senders -> produce side-effects

Adapters isolate integrations:

src/adapters/<name>/receiver/
src/adapters/<name>/sender/

Adding a new integration means adding an adapter, not touching core logic.

Current adapters

Dashboard (src/adapters/dashboard/)

Receiver: REST API
Sender: writes to D1

Slack (src/adapters/slack/)

Receiver: events, interactions
Sender: messages, threads, channels

Status page (src/dispatcher/status-page.ts)

Sender-only: persists affections to external DB

Each adapter handles idempotency in its own way.

The AI agent

Each incident runs an agent that watches the timeline and generates suggestions.

It helps:

resolve the incident
keep stakeholders informed

The agent runs asynchronously.
It reads from the timeline and writes back into it.

It is more inspired by Codex than by a classic tool-calling loop: each turn is a IncidentAgentTurnWorkflow run.

Debouncing

Not every event should trigger a turn.

First turn: delayed 60s (let the incident stabilize)
Subsequent turns: 13s debounce window

Execution

Each turn:

Fetch snapshot (state + events)
Call LLM with tools
Optionally trigger context agents
Persist suggestions as events
Post to Slack

Suggestions are stored as internal events, so future turns have memory and avoid repetition.

Context agents

Context agents are separate Durable Objects that run background investigations.

That keeps the incident DO focused on coordination, not research.

They are decoupled to:

run asynchronously
avoid polluting the main incident context

Base agent

Context agents extend AgentBase, which provides:

SQLite storage -> (steps, contexts)
Summarization -> keeps token usage bounded
Alarm-driven execution
Lazy initialization

Bidirectional RPC

The incident DO and context agents communicate in both directions:

-> provider:

addContext: pushes new timeline events for background processing
addPrompt: handles user questions from Slack, pulling the provider’s latest findings

Provider ->:

getAgentContext: pulls the full incident snapshot to inform investigation
recordAgentInsightEvent: writes insight events (SIMILAR_INCIDENT, GITHUB_COMMIT) without published_at, so they flow through the normal alarm/dispatch cycle and trigger new agent turns

Current providers

SimilarIncidentsAgent -> searches past incidents for similar patterns.
GitHubCommitsAgent -> surfaces relevant commits

Future ones:

AWS investigator (via CLI, on Container-enabled Durable Objects)
DataDog (via MCP)

Prompt workflow

Users can directly ask the agent questions. Handled by IncidentPromptWorkflow:

LLM decides what to do
If needed, calls context agents
Executes action or responds

A :fire: reaction shows progress.

Expected usage is command-like (“resolve incident”, “post status page update”), making it easy for untrained responders.

Reliability

Everything optimizes for one goal:

never lose an event

Acknowledgement happens after persistence
State + event are atomic
Delivery is retried
Senders are isolated

So far: ** 0 lost events**, with low double-digit millisecond latency.

Folder structure

- Adapters: src/adapters/*
- Handler: src/handler
- Core DO: src/core/incident.ts
- Alarm: src/core/incident/alarm.ts
- Dispatcher: src/dispatcher/workflow.ts
- Agents: src/agent/providers/

The only part I’m still unsure about is how context agents will interact as their number grows.

We’ll find out.

Worker configuration (Node) > Deep dive

Thu, 08 Oct 2026 13:07:43 GMT

Worker configuration (Node) > Deep dive

This article is a deep dive into the worker configuration and the many experiments I have run to understand it.
It’s intended for those who want to understand the internals of Worker configuration. If you’re looking for a simpler setup or just want to know why I ran these tests, start with the main article.

This is not a definitive guide. Some parts remain unclear, but it documents what I’ve learned so far and where the documentation or defaults can be misleading.

It focuses on:

The tuner
Poller configuration
Worker thread configuration

It does NOT focus on:

optimizing workflow/activity code
keeping history short
caching workflows / sticky vs non-sticky pollers

The tuner

The resource-based tuner aims to make Workers require almost no configuration. You specify a target CPU and memory usage, and the tuner will grant slots of each task type workflow/activity/local-activity/nexus until the target CPU and memory usage are reached.

In practice, I haven’t been able to make it behave as expected. Here’s what I found.

Under-reports CPU and memory usage

Tuner vs CloudWatch CPU and memory usage for 1 vCPU instance

Tuner vs CloudWatch CPU and memory usage for 2 vCPU instances

For both 1 and 2 vCPU instances, the tuner under-reports CPU and memory usage.
Hypothesis: it fails to read cgroup stats, and is reading the Node process stats instead.
It then also under-reports those, by giving a reading of around half of Node’s.

Tuner CPU vs Node CPU

Alternate hypothesis: the tuner might be ignoring main-thread CPU usage.

Tuner CPU vs Node CPU - ELU

ELU (event loop utilization) measures how much time the main Node thread spends in the event loop. It’s a strong proxy for main-thread CPU usage.

Slot grants ignore task type cost

The issue ties to the Worker threading model.

It’s NOT relevant in single-vCPU instances (where the main and worker threads share the same core) but becomes noticeable in multi-core setups.

It depends on how expensive workflow vs activity tasks are, (Considering rampThrottle and minSlots configuration) you can see three possible behaviors:

a) Activity tasks are ‘more expensive’ than workflow tasks

In this scenario, a backlog of activity tasks would accumulate. The main thread is constantly scheduled on a core, yet the CPU usage would not reach high values as the worker thread is idle.
This causes the tuner to keep granting more activity task slots, which worsens the situation.

Burst of tasks

→

Backlog accumulates

↓

Tuner grants more slots

←

CPU below target

(worker thread idle)

↓

↑

Main thread saturated

b) Workflow tasks are ‘more expensive’ than activity tasks

This is a similar scenario to the previous one, but in reverse, a backlog of workflow tasks would accumulate.

It can be solved by increasing the workflowThreadPoolSize to match the number of cores, as if all worker threads are busy, CPU will reach the target, even if the main thread is idle.

Stats comparison when increasing worker threads from 1 to 2

These images show the stats when running a workload saturated due to workflow tasks on fixed-sized tuner, going from 1 worker thread and 12 slots to 2 worker threads and 18 slots for workflow tasks and 24 slots for activity tasks.

In the graphs, CPU plateaus near 80% with one worker thread (≈ main thread at 100% + worker thread at 50%) / 2.
Adding another thread (and more workflow slots) pushes CPU to 100%, confirming full utilization.

c) Both are equally ‘expensive’

Even though this scenario is unrealistic, in this case the tuner performs well, slot distribution is balanced and increases until the target is reached.

Not granting slots? Pollers?

Given my workload is workflow-heavy, I tried setting a fixed size for workflow slots (to make sure not too many are picked and time out), and let the tuner manage activity tasks.

For some reason, activity task slots used were low.

An explanation could be not enough pollers. So I also tested:

Increasing pollers from 10 → 25 → 40
Switching from fixed-size to PollerBehaviorAutoscaling

Activity tuner with 8 workflow tasks

Activity tuner with 12 workflow tasks

Activity tuner with 6 workflow tasks and 40 pollers

None of these changed the number of activity task slots in use.

When using a fixed-size number of activity task slots, more task slots were in use, meaning the tuner likely failed to grant slots.

Fixed-size configuration: 8 workflow tasks, 24 activity tasks

Conclusion

If the tuner measured resources correctly, it would work fine on 1 vCPU instances.

On 2 vCPU instances, it could be useful for workflow-heavy workloads, assuming a fixed-size workflow task slot supplier and as many Worker threads as cores.

I haven’t succeeded in using it reliably, but I’ve learned about the tuner, pollers, the threading model and how Node interacts with all of these.

My current configuration (might go back to 1vCPU with a fixed-size workflow task slot supplier, which is simpler):

2 vCPUs
2 worker threads
14 slots for workflow tasks
24 slots for activity tasks

Since we don’t fan out into many activities, activity slot count doesn’t matter much as long as there are enough slots. (24 or 24,000 would work just as well)

Appendix

If you can shed light on any of the unexplained behavior, please add a comment below or reach out at miquel@miquelpuigturon.com or on Temporal Slack (Miquel).

I have more graphs and data available for anyone interested.

Additional data and images from experiments

Cached WFs	Heap (MB)	RSS (MB)	Native (MB)	Delta/10s (MB)
0	195	501	285	—
4	192	535	322	+34
29	199	543	322	+5
78	214	638	386	+2
92	189	593	376	+1

Cached WFs	Heap (MB)	RSS (MB)	Native (MB)	Delta/10s (MB)
0	195	501	285	—
4	192	535	322	+34
29	199	543	322	+5
78	214	638	386	+2
92	189	593	376	+1

Miquel Puig Turon

Humanity

Humanity

Self

Self

Welcome

Welcome

Why

What

Where to start

Fire

Fire

Money

Money

How money grows

Why non-asset owners won’t be rich

Why inheritance sucks

Tres mesos intentant construir software per a farmàcies

Tres mesos intentant construir software per a farmàcies

Per què farmàcia

El mercat de la farmàcia

Volum i marge

Diferenciació i serveis

Orkava

Què hem après

Els competidors

Oportunitats

Conclusió

Temporal

Temporal

Addictions

Addictions

Why I avoid them

Desire for greatness

Innocent belief in perfection

Novelty

Some not-so-obvious addictions

Porn

Coffee

Love

Digital Consumption

Fighting addictions

Avoiding temptation

Self-honesty

Compounding effect

How I Want to Live My Life

How I Want to Live My Life

Why am I writing this

How my life has been up to now

Physical Effort

Coding

Happiness

How I will do it

Dream state

Meta

Meta

Philosophy

Technical Stack

Content Structure

Automatic Routing

Getting Around

Design Choices

Source

2026

2026

Goals

Sport

Sport

Past

Present

Future

WebMCP

WebMCP

Tools available to browser agents

How it works

For agents without browser WebMCP

Status

Pain, Suffering and God

Pain, Suffering and God

Pain

Cached WFs	Heap (MB)	RSS (MB)	Native (MB)	Delta/10s (MB)
0	195	501	285	—
4	192	535	322	+34
29	199	543	322	+5
78	214	638	386	+2
92	189	593	376	+1