Worker configuration (Node) > Deep dive

This article is a deep dive into the worker configuration and the many experiments I have run to understand it.
It’s intended for those who want to understand the internals of Worker configuration. If you’re looking for a simpler setup or just want to know why I ran these tests, start with the main article.

This is not a definitive guide. Some parts remain unclear, but it documents what I’ve learned so far and where the documentation or defaults can be misleading.

The tuner

The resource-based tuner aims to make Workers require almost no configuration. You specify a target CPU and memory usage, and the tuner will grant slots of each task type workflow/activity/local-activity/nexus until the target CPU and memory usage are reached.

In practice, I haven’t been able to make it behave as expected. Here’s what I found.

Under-reports CPU and memory usage

For both 1 and 2 vCPU instances, the tuner under-reports CPU and memory usage.
Hypothesis: it fails to read cgroup stats, and is reading the Node process stats instead.
It then also under-reports those, by giving a reading of around half of Node’s.

Alternate hypothesis: the tuner might be ignoring main-thread CPU usage.

ELU (event loop utilization) measures how much time the main Node thread spends in the event loop. It’s a strong proxy for main-thread CPU usage.

Slot grants ignore task type cost

The issue ties to the Worker threading model.

It’s NOT relevant in single-vCPU instances (where the main and worker threads share the same core) but becomes noticeable in multi-core setups.

It depends on how expensive workflow vs activity tasks are, (Considering rampThrottle and minSlots configuration) you can see three possible behaviors:

a) Activity tasks are ‘more expensive’ than workflow tasks

In this scenario, a backlog of activity tasks would accumulate. The main thread is constantly scheduled on a core, yet the CPU usage would not reach high values as the worker thread is idle.
This causes the tuner to keep granting more activity task slots, which worsens the situation.

b) Workflow tasks are ‘more expensive’ than activity tasks

This is a similar scenario to the previous one, but in reverse, a backlog of workflow tasks would accumulate.

It can be solved by increasing the workflowThreadPoolSize to match the number of cores, as if all worker threads are busy, CPU will reach the target, even if the main thread is idle.

These images show the stats when running a workload saturated due to workflow tasks on fixed-sized tuner, going from 1 worker thread and 12 slots to 2 worker threads and 18 slots for workflow tasks and 24 slots for activity tasks.

In the graphs, CPU plateaus near 80% with one worker thread (≈ main thread at 100% + worker thread at 50%) / 2.
Adding another thread (and more workflow slots) pushes CPU to 100%, confirming full utilization.

c) Both are equally ‘expensive’

Even though this scenario is unrealistic, in this case the tuner performs well, slot distribution is balanced and increases until the target is reached.

Not granting slots? Pollers?

Given my workload is workflow-heavy, I tried setting a fixed size for workflow slots (to make sure not too many are picked and time out), and let the tuner manage activity tasks.

For some reason, activity task slots used were low.

An explanation could be not enough pollers. So I also tested:

• Increasing pollers from 10 → 25 → 40
• Switching from fixed-size to PollerBehaviorAutoscaling

None of these changed the number of activity task slots in use.

When using a fixed-size number of activity task slots, more task slots were in use, meaning the tuner likely failed to grant slots.

Conclusion

If the tuner measured resources correctly, it would work fine on 1 vCPU instances.

On 2 vCPU instances, it could be useful for workflow-heavy workloads, assuming a fixed-size workflow task slot supplier and as many Worker threads as cores.

I haven’t succeeded in using it reliably, but I’ve learned about the tuner, pollers, the threading model and how Node interacts with all of these.

My current configuration (might go back to 1vCPU with a fixed-size workflow task slot supplier, which is simpler):

• 2 vCPUs
• 2 worker threads
• 14 slots for workflow tasks
• 24 slots for activity tasks

Since we don’t fan out into many activities, activity slot count doesn’t matter much as long as there are enough slots. (24 or 24,000 would work just as well)

Appendix

If you can shed light on any of the unexplained behavior, please add a comment below or reach out at miquel@miquelpuigturon.com or on Temporal Slack (Miquel).

I have more graphs and data available for anyone interested.