Worker configuration (Node)

This is a walkthrough of my experience configuring workers running on Node for Genesy.

It focuses on worker configuration, as I found the current (as of Oct 2025) defaults to be misguiding, and want to share my findings, as this is not only a me-issue.

Readers should understand workers and worker/server interaction before reading this article. (a good resource)

Fun fact: This article is the reason this site exists.

Background

Click to expand

We run two services Each service simply runs a worker

• Enrich: User actions are not processed in the API directly, they are off-loaded to this service, which makes sure that they are processed reliably. I wrote my CS bachelor’s thesis on why we decided on this, and how I coded the primitives that allow it to work well.
• Conversation: Genesy offers automated campaigns, which are likely the most advanced conditional campaign system that exist on any tool. Each campaign has an identity which targets contacts, with which a conversation is started. Each conversation runs as an entity workflow, and identity actions are centralized in a manager workflow.

Even though the learnings apply to both services, most of the investigation comes from the Conversation service, as we want it to scale to infinity reliably (started in February with <50k active conversations, in October it handles >500k).

Why did I start looking into this

Users can see conversations in Genesy via the inbox. The inbox also allows sending messages directly.

There’s nothing worse for users than sending a message twice. To avoid this, we use updates to prompt the manager workflow to send the message, and get a response (whether it was successful or not).

If the manager workflow is not responsive, then the update fails, and the user is sad. As my main job is to make users happy, this had to be solved.

Priority

6th of march, Temporal announced priority and fairness.

It took 3 months of me complaining that it wasn’t working, until priority was an actual useful feature.

I thought that priority would solve this problem. Simply prioritize manager workflows over conversation workflows. Then we only need to make sure that we can handle hundreds of workflows immediately instead of hundreds of thousands.

Discovering the problem

I started experimenting with how to make sure priorityKey=1 workflows are always responsive no matter how many higher priority workflows needed processing. An easy way to stress test it was to limit the number of instances of the service, and thus of workers.

Previously, up to 50 instances were running at a time, as our best guess on making priorityKey=1 workflows responsive was to process everything instantly.
I cut down to 10 instances, and at peak times, workers were saturated By saturated I mean that for the task queue, the backlog increases: tasksAddRate > tasksDispatchRate .

Even though it looks great, as the backlog age is less than one second for priorityKey=1, it performed worse.
That day, we went from ~97% success on users sending messages to ~85%.

At this point, tasks were picked up by workers almost instantly, yet they failed. Why??
Looking at metrics, many workflow tasks were failing due to status_code=not_found

It’s obvious from the image that the failures are related to the task execution latency Time since the worker picks up the task until it responds to the temporal server . kappa agreed:

metric temporal_request_failure with operation=RespondWorkflowTaskCompleted and
status_code=NOT_FOUND indicates that a worker attempted to report the completion
of a workflow task, but the Temporal service could not find the corresponding
workflow task. This typically happens when the workflow task has already timed
out—meaning the worker did not respond within the default 10-second timeout window

The worker was taking more than 10 seconds (default start-to-close timeout) to complete the task.
This was a surprise, as workflow tasks are lightweight, and should take tens of milliseconds at most.

Understanding worker configuration

The worker configuration is defined on the worker.create method. Here we focus on performance-related options.

Task execution and polling

Each worker has a number of pollers and slots for each task type workflow/activity/local-activity/nexus . The pollers grab tasks from the Temporal server, and occupy a slot while the task is being executed. The reasoning behind having multiple pollers and not just one is due to performance. From the code docs:

In general, a Workflow Worker's performance is mostly network bound (due to
communication latency with the Temporal server). Accepting multiple Workflow
Tasks concurrently helps compensate for network latency, until the point
where the Worker gets CPU bound.

There are two ways to configure this:
The original way, using maxConcurrent <taskType> workflow/activity/local-activity/nexus task Executions and maxConcurrent <taskType> workflow/activity/local-activity/nexus task Polls.
The new way, using the tuner, which grants slots until a targetCpu and targetMemory are reached.

We will focus on workflow and activity tasks, which are the majority of tasks that are executed.

Threading model

The threading model is explained here.
Basically, interactions with the Core SDK happen on the main thread (Node’s event loop). There’s also at least one worker thread, which the main thread delegates workflow tasks to. This is a great idea, as workflow tasks are CPU bound (they run until the next await).

The number of worker threads is configured with the workflowThreadPoolSize option. It defaults to 1 if reuseV8Context is true, or 2 otherwise.

Workflow tasks tend to be lightweight, so the defaults are usually fine.

Sticky tasks and worker cache

There’s also the nonStickyToStickyPollRatio, which indicates the ratio of pollers that will poll from the sticky task queue.
Along with maxCachedWorkflows, it is a very useful optimization to avoid having to replay workflows.

The values should be set according to your workload patterns.
The docs on it are good, so no need to go deeper here.

What can we do to fix the problem?

Task execution and polling

By far the most relevant configuration is related to task execution and polling.

Looking at metrics for number of slots slots are tasks being executed concurrently , we can see that the number of workflow slots is directly related to the number of tasks failing, and through the previous relationship, the number of workflow tasks timing out.

This image is obtained using the tuner, targeting 0.8 CPU and 0.6 memory usage, with a workflow task rampThrottle time before handling new slots of 100ms. Without the rampThrottle, we would get a lot of “out of memory” errors, as the worker would fill its memory trying to execute too many tasks at once.

It’s obvious from the image that the tuner is granting too many slots, and that this for some reason, is causing tasks to fail.

The worker takes more than the start-to-close timeout to complete the tasks, so when responding to the Temporal server with a result, the task is no longer found, and the result is discarded. This is very bad, as that same task is reissued, and the process repeats, burning CPU cycles for no reason.

Threading model

In all my tests, the tuner grants way too many workflow slots no matter what targetCPU is specified. To understand why, we need to look at the threading model.

This is running on ECS Fargate, on single vCPU instances.

Running 2 vCPU instances, with the default configuration, the tuner was slightly more useful, but still granted too many slots.

When configuring a fixed sized workflow task slot supplier, with 2 vCPUs and one worker thread, I could get a bit less than twice the number of slots as with 1 vCPU, but maximum CPU usage was less than 75%, which makes sense, the worker thread was at full capacity, and the main thread was half-idle, mostly due to I/O bound work.

Conclusions

I couldn’t get the tuner to work as expected, so I can’t recommend using it, it causes too many workflow task slots, which makes them time out and be retried, burning CPU cycles for no reason.

The most important thing is to use a fixed sized workflow task slot supplier, with a size that is a good fit for your workload.

If no configuration is used (no tuner, no maxConcurrentWorkflowTaskExecutions), the worker will use the default configuration, which is 40 workflow slots (likely still way too many).

Why is the default worker configuration so bad?

Well, It’s not that bad…
For parallel programming languages (like go and java, which are likely Temporal’s target languages), it makes sense, as we want to reduce the percentage of I/O work, and there’s no event loop or main thread that limits performance.

I have explored more in detail the tuner, running on 2vCPU instances, and poller configuration, which you can read about in the deep-dive article.

A decent worker configuration

We now use A good enough configuration is the following:

const config = {
    ... other things
    tuner = {
        workflowTaskSlotSupplier: {
            type: 'fixed-size',
            numSlots: X (5 for conversation, 10 for enrich),
        },
        activityTaskSlotSupplier: {
            type: 'resource-based',
            rampThrottle: '100 ms',
            tunerOptions: {
                targetCpuUsage: 1.0,
                targetMemoryUsage: 0.6,
            },
        },
        // local and nexus tasks don't really matter
    },
    nonStickyToStickyPollRatio: Y (2/5 for conversation, 0.5 for enrich),
    maxCachedWorkflows: Z (150 for conversation, 300 for enrich),
}

The difference between conversation and enrich is that conversation workflow tasks are slower, due to replay, and more of the conversation workflows end up not being sticky.

Values for activity tasks are not very relevant, as our workflows don’t fan out into many activities, and eager dispatch is enabled.
Activities are mostly processed instantly, and are very cheap, though they do sit in the event loop, adding a bit of latency to the main thread.

Running the same load, we can see that with 6 slots, there are no failures, and that the maximum time that workflow tasks take to process is close to the start-to-close timeout. With this configuration, 100% of users sending messages succeed.

Why not just 1 slot?

Even though the default is not good, the docs are correct in that worker performance is mostly network bound so it’s still useful to have multiple slots. We want to make sure that either the workflow thread or the main thread are never idle due to having no work to do.

Running workers in production

This article is related to worker configuration, but to run workers in production, infrastructure and scaling strategies also need to be considered.

Workers should run on fixed-size instances and scale horizontally. (We use ECS Fargate, and run 1-50 (usually under 10) instances per service, larger teams probably use Kubernetes)

Scaling should be done on average CPU usage across worker instances, and either task backlog count or backlog age. (We send task queue metrics to CloudWatch)

Further reading

If you are interested in understanding the configuration and the details of the many experiments I have run, you can read the deep-dive article.