When you start an Erlang or Elixir system, you get an OS process running the Erlang Runtime System, and inside that OS process, the BEAM VM is executing a number of Erlang processes. Such a running instance of the Runtime System is referred to as an Erlang node (regardless of whether it runs ERTS or another implementation of Erlang — see Section 1.4), and it can be given a name to distinguish it from other Erlang nodes that may be running in the same network or even on the same host machine. The equivalent in the Java world is a JVM instance; however, Erlang also has a built-in concept of transparent distribution, and Erlang nodes can be connected to each other in a cluster across the network.

Your BEAM application code will thus always be executing within the context of an Erlang node, and all the layers of the node will affect the performance of your application. We will look at the stack of layers that makes up a node. This will help you understand your options for running your system in different environments.

To be completely correct according to the Erlang OTP documentation, a node is actually an executing runtime system that has been given a name and is ready to participate in a cluster. That is, if you start Elixir without giving a name through one of the command line switches --name NAME@HOST or --sname NAME (or -name and -sname for an Erlang runtime) you will strictly speaking have a runtime but not a node. In such a system the function Node.alive? (or in Erlang is_alive()) returns false.

The runtime system itself is not that strict in its use of the terminology. You can ask for the name of the node even if you didn’t give it a name. In Elixir you use the function Node.list with the argument :this, and in Erlang you can call nodes(this) or just node():

In this book we will use the term node for any running instance of the runtime whether it is given a name or not.

In Part I of this book you will learn all about the components of ERTS. In Part II, we describe how to run, inspect, debug, and profile an Erlang node. For information on how to build Erlang from source, see Appendix A.

Your application(s) will run in one or more nodes, and the performance of your program will depend not only on your application code but also on all the layers below your code in the ERTS stack. In Figure 1 you can see the ERTS Stack illustrated with two Erlang nodes running on one machine.

If you are using Elixir there is yet another layer to the stack.

Let’s look at each layer of the stack and see how you can tune them to your application’s need.

At the bottom of the stack there is the hardware you are running on. The easiest way to improve the performance of your app is probably to run it on better hardware. You might need to start exploring higher levels of the stack if economical or physical constraints or environmental concerns won’t let you upgrade your hardware.

The two most important choices for your hardware is whether it is multicore and whether it is 32-bit or 64-bit. You need different builds of ERTS depending on whether you want to use multicore or not and whether you want to use 32-bit or 64-bit.

The second layer in the stack is the OS level. ERTS runs on most versions of Windows and most POSIX "compliant" operating systems, including Linux, FreeBSD, Solaris, and Mac OS X. Today most of the development of ERTS is done on Linux and OS X, and you can expect the best performance on these platforms. However, Ericsson has been using Solaris internally in many projects and ERTS has been tuned for Solaris for many years. Depending on your use case you might actually get the best performance on a Solaris system. The OS choice is usually not based on performance requirements, but is restricted by other factors. If you are building an embedded application you might be restricted to Raspbian and if you for some reason are building an end user or client application you might have to use Windows. The Windows port of ERTS has so far not had the highest priority and might not be the best choice from a performance or maintenance perspective. If you want to use a 64-bit ERTS you of course need to have both a 64-bit machine and a 64-bit OS. We will not cover many OS specific questions in this book, and most examples will be assuming that you run on Linux.

The third layer in the stack is the Erlang Runtime System. In our case this will be ERTS. This and the fourth layer, the Erlang Virtual Machine (BEAM), is what this book is all about.

The fifth layer, OTP, supplies the Erlang standard libraries. OTP originally stood for "Open Telecom Platform" and was a number of Erlang libraries supplying building blocks (such as supervisor, gen_server and gen_tcp) for building robust applications (such as telephony exchanges). Early on, the libraries and the meaning of OTP got intermingled with all the other standard libraries shipped with ERTS. Nowadays most people use OTP together with Erlang in "Erlang/OTP" as the name for ERTS and all Erlang libraries shipped by Ericsson. Knowing these standard libraries and how and when to use them can greatly improve the performance of your application. This book will not go into any details of the standard libraries and OTP, there are many other books that cover these aspects.

If you are running an Elixir program the sixth layer provides the Elixir environment and the Elixir libraries.

Finally, the seventh layer (Apps) is your applications, and any third party libraries you use. The applications can use all the functionality provided by the underlying layers. Apart from upgrading your hardware this is probably the place where you most easily can improve your application’s performance. In Chapter 21 there are some hints and some tools that can help you profile and optimize your application. In Chapter 18 we will look at how to find the cause of crashing applications and how to find bugs in your application.

One of the key insights by the Erlang language designers was that in order to build a system that works 24/7 you need to be able to handle hardware failure. Therefore you need to distribute your system over at least two physical machines. You do this by starting a node on each machine and then you can connect the nodes to each other so that processes can communicate with each other across the nodes just as if they were running in the same node. See Chapter 14 for details about distribution.

The Erlang Compiler is responsible for compiling Erlang source code, from .erl files into BEAM virtual machine code. The compiler is itself an Erlang application, written in Erlang and compiled by itself into BEAM code. In a shipped Erlang-based system (a release), the compiler might or might not be included depending on whether that system needs to be able to compile code as well as run it.

To bootstrap the runtime system, the source distribution of Erlang includes a number of precompiled BEAM files, including the compiler, in the bootstrap and erts/preloaded directories.

For more information about the compiler see Chapter 2.

BEAM is the virtual machine used for executing Erlang code, just like the JVM is used for executing Java code. BEAM code runs in the context of an Erlang Node.

Just as ERTS is an implementation of a more general concept of a Erlang Runtime System so is BEAM an implementation of a more general Erlang Virtual Machine (EVM). There is no definition of what constitutes an EVM but BEAM actually has two levels of instructions: Generic Instructions and Specific Instructions. The generic instruction set could be seen as a blueprint for an EVM.

A detailed description of the BEAM can be found in Chapter 5 and the following chapters.

An Erlang process basically works like an OS process. Each process has its own private memory (a mailbox, a heap and a stack) and a process control block (PCB) with information about the process. Erlang processes are not like threads — they have no shared memory to modify, they can only communicate via messages, and from one process' point of view there is very little difference if another process runs in the same VM or is running on another node connected over the network. Erlang programs do not need locking or protected sections.

All Erlang code execution is done within the context of a process. One Erlang node can have many processes, which communicate through message passing and signals. Erlang processes can also communicate with processes on other Erlang nodes as long as the nodes are connected.

To learn more about processes and the PCB see Chapter 3.

The Scheduler is responsible for choosing the Erlang process to execute. Basically the scheduler keeps two queues, a ready queue of processes ready to run, and a waiting queue of processes waiting to receive a message.

The scheduler picks the first process from the ready queue and hands it to BEAM for execution of one time slice. BEAM preempts the running process when the time slice is used up and moves the process back to the end of the ready queue. If the process is blocked in a receive before the time slice is used up, it gets moved to the waiting queue instead. When a process in the waiting queue receives a message or gets a time out, it is moved to the ready queue.

Erlang is concurrent by nature, that is, each process is conceptually running at the same time as all other processes, but in reality the scheduler is just running one process at a time. Hence, even on a single core machine, and even if the underlying OS does not have preemption, the BEAM is still capable of running the same Erlang programs with hundreds of thousands of concurrent processes.

On a multicore machine, Erlang will automatically run more than one scheduler, in separate OS threads — usually one per physical core, each having their own run queues. If one scheduler runs out of work, it can take over some of the ready processes from the other schedulers. This way Erlang achieves true parallelism, without the programmer having to worry about it.

In reality the picture is more complicated with priorities among processes and the waiting queue is implemented through a timing wheel. All this and more is described in detail in Chapter 10.

Erlang is a dynamically typed language, and the runtime system needs a way to keep track of the type of each data object. This is done with a tagging scheme. Each data object or pointer to a data object also has a tag with information about the data type of the object.

Basically, some bits of a pointer are reserved for the tag, and the VM can then determine the type of the object by looking at the bit pattern of the tag.

These tags are used for pattern matching and for type tests and for primitive operations, as well as by the garbage collector.

The complete tagging scheme is described in Chapter 4.

Erlang uses automatic memory management and the programmer does not have to think about memory allocation and deallocation. Each process has a heap and a stack, and these can both grow and shrink as needed. There are no stack size limits to worry about, and no preallocated address ranges for stacks, as there can be for OS threads. Initially, both the stack and the heap for a process are quite small, which allows you to have many thousands or even millions of processes.

When a process runs out of heap space, the VM will first try to reclaim free heap space through garbage collection. The garbage collector will then go through the process stack and heap and copy live data to a new heap while throwing away all the data that is dead. If there still isn’t enough heap space, a new larger heap will be allocated and the live data is moved there.

The details of the current generational copying garbage collector, including the handling of reference counted binaries can be found in Chapter 11.

When you start an Erlang node with erl you get a command prompt. This is the Erlang read eval print loop (REPL) or the command line interface (CLI) or simply the Erlang shell.

You can type in Erlang expressions and execute them directly from the shell. In this case the code is not compiled to BEAM code and executed by the BEAM. Instead, the code is parsed and interpreted by the Erlang interpreter. In general, the interpreted code behaves exactly as compiled code, but there are a few subtle differences — in particular, the interpreted code is slower. These differences and all other aspects of the shell are explained in Chapter 17.

Erlang on Xen (https://github.com/cloudozer/ling) is an Erlang implementation running directly on server hardware with no OS layer in between, only a thin Xen client.

Ling, the virtual machine of Erlang on Xen is almost 100% binary compatible with BEAM. In Figure 4 you can see how the Erlang on Xen implementation of the Erlang Solution Stack differs from the ERTS Stack. The thing to note here is that there is no operating system in the Erlang on Xen stack.

Since Ling implements the generic instruction set of BEAM, it can reuse the BEAM compiler from the OTP layer to compile Erlang to Ling.

Erjang (https://github.com/trifork/erjang) is an Erlang implementation which runs on the JVM. It loads .beam files and recompiles the code to Java .class files. Erjang is almost 100% binary compatible with (generic) BEAM.

In Figure 5 you can see how the Erjang implementation of the Erlang Solution Stack differs from the ERTS Stack. The thing to note here is that JVM has replaced BEAM as the virtual machine and that Erjang provides the services of ERTS by implementing them in Java on top of the JVM.

Now that you have a basic understanding of all the major pieces of ERTS, and the necessary vocabulary, you can dive into the details of each component. If you are eager to understand a certain component, you can jump directly to that chapter. Or if you are really eager to find a solution to a specific problem, you could jump to the right chapter in Part II and try the different methods to tune, tweak, or debug your system.

The compilation starts with tokenization (or scanning) and preprocessing. The preprocessor epp reads a file and calls the tokenizer erl_scan to expand the text into a sequence of separate tokens rather than characters, discarding whitespace and comments. It then processes macro definitions and conditional compilation directives, and substitutes uses of macros.

When the preprocessor finds an include statement, it reads and tokenizes the named include file, inserts the resulting token sequence instead of the include statement, and then processes that as well so that files can include each other recursively. When it switches between one file and another, the preprocessor inserts -file annotations, as we saw in the example above, so that later passes can know where a particular piece of code came from:

Note that when an include file like world.hrl gets processed, there may be no actual code inserted, since include files often contain only macro definitions and preprocessor conditionals. All that is left are the annotations saying “we are now at line 1 of the header file” and “we are now back at line 4 in the original source file”.

Note that all this means that epp works on the token level (just like the C/C++ preprocessor cpp), so it is not a pure string replacement processor (as for example m4). You cannot use Erlang macros to define your own arbitrary syntax; a macro will expand as a separate token from its surrounding characters, so you can’t concatenate a macro and a character to a token:

This will expand to

and not

On the other hand since macro expansion is done on the token stream before the actual parsing, you do not need to have a valid Erlang term in the right hand side of the macro, as long as you use it in a way that gives you a valid term in the end. E.g.:

which will expand to the token sequence

There are few real usages for this other than to win the obfuscated Erlang code contest. The main point to remember is that you can’t really use the Erlang preprocessor to define a language with a syntax that differs from Erlang.

The parser erl_parse gets the final token sequence from the preprocessor and checks it against the Erlang grammar, producing an abstract syntax tree or AST (see http://www.erlang.org/doc/apps/erts/absform.html ) representing the program as a data structure instead of as a sequential text.

A parse transform is a function that works on the AST. After the compiler has done the initial tokenization, preprocessing and parsing, and assuming there were no errors so far, it will call the parse transform function if one has been declared, passing it the current AST and expecting to get a modified AST back.

This means that you can’t change the Erlang syntax fundamentally, because the input must still be accepted by the Erlang parser, but you can change the semantics by modifying the code as you like.

A parse transform is declared like this, in the module where it is to be used:

The compiler will then try to call my_pt:parse_transform(Forms, Options) as an extra pass. It is possible to use multiple parse transforms in the same module; they will simply be executed in order.

Parse transforms are used by libraries such as Mnesia (for its QLC syntax), EUnit, Merl, and others. In Section 2.5 we will demonstrate a full example of how you can implement your own parse transform.

The term “linter” originally meant a tool that was an add-on to a compiler — like a lint filter in a clothes dryer — checking for stylistic errors and antipatterns and other things that the compiler itself did not check (because in those days compilers had very little memory available and were focused on speed). As computers got faster, however, the linter got moved into the compiler itself as a pass that you wanted to run all the time, not just occasionally.

In the Erlang compiler, the linter erl_lint is the stage that does most of the checking of the source code, after parsing and transforms are done. (This means that parse transforms may only return an AST that the linter will accept.)

The linter checks for errors, such as undefined variables and functions, as well as generating warnings for correct but suspicious code, such as "export_all flag enabled".

In order to enable debugging of a module you must “debug compile” the module, that is pass the option debug_info to the compiler. The abstract syntax tree will then be saved by the “Save AST” pass, until the end of the compilation where it will be included in the .beam file for purposes of setting breakpoints, single stepping etc. The code is then running interpreted, not compiled.

It is important to note that the code is saved before any optimisations are applied, so if there is a bug in an optimisation pass in the compiler and you run interpreted code in the debugger you will get a different behavior. If you are implementing your own compiler optimisations, this can trick you up badly.

In the expand phase, source level Erlang constructs such as records are expanded to lower level erlang constructs. This strips compiler directives, replaces record syntax with operations on tuples, expands uses of function imports to explicit remote calls, and adds the built in functions module_info/{1,2} to the code.

Core Erlang is a strict functional language suitable for compiler optimizations. It makes code transformations easier by reducing the number of ways to express the same operation. One way it does this is by introducing let and letrec expressions to make scoping more explicit. The compiler does several passes on the core level.

Core Erlang can be a good target for a language you want to run in ERTS. It changes very seldom and it contains all aspects of Erlang in a clean way. If you target the BEAM instruction set directly, you will have to deal with many more details, and that instruction set usually changes slightly between each major release of ERTS. If you on the other hand target Erlang directly you will be more restricted in what you can describe, and you may also have to deal with more details, since Core Erlang is a cleaner language.

Note however that targeting plain Erlang gives you more compile time checking that your generated code is well behaved. By generating Core Erlang directly, you may manage to find corner cases that the Erlang compiler never has encountered before; there is a greater responsibility on you to generate correct code.

To compile an Erlang file to core you can give the option to_core. This outputs the resulting Core Erlang program to a file with the extension .core. To compile a Core Erlang program from a .core file you can give the option from_core to the compiler.

Note that the .core files are text files written in the human readable core format. To get the core program as an Erlang term you can add the binary option to the compilation.

Kernel Erlang is a flat version of Core Erlang with a few differences. For example, each variable is unique and the scope is a whole function. Pattern matching is compiled to more primitive operations. The kernel representation does not have a well defined file format and you should not expect it to be stable.

The last stage of a normal compilation is the external BEAM assembly code format, as a sequence of individual BEAM instructions represented as Erlang terms. Some low level optimizations such as dead code elimination and peep hole optimisations are done on this level.

The BEAM code is described in detail in Chapter 7 and Appendix B

The BEAM assembly code is finally packed into the binary transport format which can be written to a .beam file, sent over the network, or loaded directly into memory through code:load_binary/3. See Chapter 6] for details.

Leex is the Erlang lexer generator. The lexer generator takes a description of a DFA from a definitions file xrl and produces an Erlang program that matches tokens described by the DFA.

The details of how to write a DFA definition for a tokenizer is beyond the scope of this book. For a thorough explanation I recommend the "Dragon book" ([DragonBook]). Other good resources are the man and info entry for flex, the lexer program that inspired leex, and the leex documentation itself. If you have info and flex installed you can read the full manual by typing:

The online Erlang documentation also has the leex manual (see yecc.html).

We can use the lexer generator to create an Erlang program which recognizes JSON tokens. By looking at the JSON definition http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf we can see that there are only a handful of tokens that we need to handle.

By using the Leex compiler we can compile this DFA to Erlang code, and by giving the option dfa_graph we also produce a dot-file which can be viewed with e.g. Graphviz.

You can view the DFA graph (Figure 8) using for example dotty:

We can try our tokenizer on an example json file (test.json).

First we need to compile our tokenizer, then we read the file and convert it to a string. Finally we can use the string/1 function that leex generates to tokenize the test file.

The shell function f/1 tells the shell to forget a variable binding. This is useful if you want to try a command that binds a variable multiple times, for example as you are writing the lexer and want to try it out after each rewrite. We will look at the shell commands in detail in the later chapter.

Armed with a tokenizer for JSON we can now write a json parser using the parser generator Yecc.

Yecc is a parser generator for Erlang. The name comes from Yacc (Yet Another Compiler Compiler) the canonical parser generator for C. The modern open source equivalent is Bison, which is often used with Flex.

Now that we have a lexer for JSON terms we can write a parser using yecc.

Then we can use yecc to generate an Erlang program that implements the parser, and call the parse/1 function provided with the tokens generated by the tokenizer as an argument.

The tools Leex and Yecc are nice when you want to compile your own complete language to the Erlang virtual machine. By combining them with Syntax tools and specifically Merl you can manipulate the Erlang Abstract Syntax tree, either to generate Erlang code or to change the behaviour of Erlang code.

Let us dive right in and look at which processes we have in a running system. The easiest way to do that is to just start an Erlang shell and issue the shell command i(). In Elixir you can call the function in the shell_default module as :shell_default.i.

The i/0 function prints out a list of all processes in the system. Each process gets two lines of information. The first two lines of the printout are the headers telling you what the information means. As you can see you get the Process ID (Pid) and the name of the process if any, as well as information about the code where the process was started and where it is currently executing. You also get information about the heap and stack size and the number of reductions and messages in the process. In the rest of this chapter we will learn in detail what a stack, a heap, a reduction and a message are. For now we can just assume that if there is a large number for the heap size, then the process uses a lot of memory and if there is a large number for the reductions then the process has executed a lot of code.

We can further examine a process with the i/3 function. Let us take a look at the code_server process. We can see in the previous list that the process identifier (pid) of the code_server is <0.36.0>. By calling i/3 with the three numbers of the pid we get this information:

We got a lot of information from this call and in the rest of this chapter we will learn in detail what most of these items mean. The first line tells us that the process has been given a name code_server. Next we can see in which function the process is currently executing or suspended (current_function) and the name of the function in which the process started executing (initial_call).

We can also see that the process is suspended waiting for messages ({status,waiting}) and that there are no messages in the mailbox ({message_queue_len,0}, {messages,[]}). We will look closer at how message passing works later in this chapter.

The fields priority, suspending, reductions, links, trap_exit, error_handler, and group_leader control the process execution, error handling, and IO. We will look into this a bit more when we introduce the Observer.

The last few fields (dictionary, total_heap_size, heap_size, stack_size, and garbage_collection) give us information about the process memory usage. We will look at the process memory areas in detail in chapter Chapter 11.

Another, even more intrusive way of getting information about processes is to use the process information given by the BREAK menu: Ctrl+c p [enter]. Note that while you are in the BREAK state the whole node freezes.

The shell functions just print the information about the process but you can actually get this information as data, so you can write your own tools for inspecting processes. You can get a list of all processes with erlang:processes/0, and information about a specific process with erlang:process_info/1. We can also use the function whereis/1 to get a pid from a name:

By getting process information as data we can write code to analyze or sort the data as we please. If we grab all processes in the system (with erlang:processes/0) and then get information about the heap size of each process (with erlang:process_info(P, total_heap_size)) we can then construct a list with pid and heap size and sort it on heap size:

You might notice that many processes have a heap size of 233 words, that is because it is the default starting heap size of a process.

See the documentation of the module erlang for a full description of the information available from process_info/2: https://erlang.org/doc/apps/erts/erlang.html#process_info/2 Notice how the process_info/1 function only returns a subset of all the information available for the process and how the process_info/2 function can be used to fetch extra information. As an example, to extract the backtrace for the code_server process above, we could run:

See the three dots at the end of the binary above? That means that the output has been truncated. A useful trick to see the whole value is to wrap the above function call using the rp/1 function:

An alternative is to use the io:put_chars/1 function, as follows:

Due to its verbosity, the output for commands 4> and 6> has not been included here, but feel free to try the above commands in your Erlang shell.

A third way of examining processes is with the Observer. Observer is an extensive graphical interface for inspecting the Erlang Runtime System. We will use the Observer throughout this book to examine different aspects of the system.

For now, we will just start Observer from the Erlang shell like this:

or from the Elixir shell with :observer.start. For debugging a live system, you probably want to launch a temporary Erlang node just for running Observer, using the -hidden flag to prevent the inspected nodes from seeing the observer node:

When Observer is started it will show you a system overview, see the following screen shot:

We will go over some of this information in detail later in this and the next chapter. For now we will just use Observer to look at the running processes. First we take a look at the Applications tab which shows the supervision tree of the running system:

Here we get a graphical view of how the processes are linked. This is a very nice way to get an overview of how a system is structured. You also get a nice feeling of processes as isolated entities floating in space connected to each other through links.

To actually get some useful information about the processes we switch to the Processes tab:

In this view we get basically the same information as with i/0 in the shell. We see the pid, the registered name, number of reductions, memory usage and number of messages and the current function.

We can also look into a process by double clicking on its row, for example on the code server, to get the kind of information you can get with process_info/2:

We will not go through what all this information means right now, but if you keep on reading all will eventually be revealed.

Now that we have a basic understanding of what a process is and some tools to find and inspect processes in a system, we are ready to dive deeper to learn how a process is implemented.

When multicore systems were introduced and the Erlang implementation was extended with more than one scheduler running processes in parallel it was no longer safe to write directly into another process' heap without taking the main lock of the receiver. At this time the concept of m-bufs was introduced (also called heap fragments). An m-buf is a memory area outside of a process heap where other processes can safely write data. If a sending process can not get the lock, it would write to the m-buf instead. When all data of a message has been copied to the m-buf, the message is linked to the process through the mailbox. The linking (LINK_MESSAGE in erl_message.h) appends the message to the receiver’s message queue.

The garbage collector would then copy the messages onto the process' heap. To reduce the pressure on the GC the mailbox is divided into two lists, one containing seen messages and one containing new messages. The GC does not have to look at the new messages since we know they will survive (they are still in the mailbox) and that way we can avoid some copying.

We can now revise our picture of the process as four memory areas once more. Now the process is made up of five memory areas (two mailboxes) and a varying number of heap fragments (m-bufs), as shown in Figure 14:

Each mailbox consists of a length and two pointers, stored in the fields msg.len, msg.first, msg.last for the internal queue and msg_inq.len, msg_inq.first, and msg_inq.last for the external in queue. There is also a pointer to the next message to look at (msg.save) to implement selective receive.

Let us use our introspection tools to see how this works in more detail. We start by setting up a process with a message in the mailbox and then take a look at the PCB.

From this we can see that there is one message in the message queue and the first, last and save pointers all point to this message.

As mentioned we can force the message to end up in the in queue by setting the flag message_queue_data. We can try this with the following program:

With this program we can try sending a message on heap and off heap and look at the PCB after each send. With on heap we get the same result as when just sending a message before:

If we try sending to a process with the flag set to off_heap the message ends up in the in queue instead:

We will ignore the distribution case for now, that is we will not consider messages sent between Erlang nodes. Imagine two processes P1 and P2. Process P1 wants to send a message (Msg) to process P2, as illustrated by Figure 15:

Process P1 will then take the following steps:

If process P2 is suspended and no other process is trying to send a message to P2 and there is space on the heap and the allocation strategy is on_heap the message will directly end up on the heap, as in Figure 16:

If P1 can not get the main lock of P2 or there is not enough space on P2 's heap and the allocation strategy is on_heap the message will end up in an m-buf but linked from the internal mailbox, as in Figure 17:

After a GC the message will be moved into the heap.

If the allocation strategy is off_heap the message will end up in an m-buf and linked from the external mailbox, as in Figure 18:

After a GC the message will still be in the m-buf. Not until the message is received and reachable from some other object on the heap or from the stack will the message be copied to the process heap during a GC.

Erlang supports selective receive, which means that a message that doesn’t match can be left in the mailbox for a later receive. And the processes can be suspended with messages in the mailbox when no message matches. The msg.save field contains a pointer to a pointer to the next message to look at.

In later chapters we will cover the details of m-bufs and how the garbage collector handles mailboxes. We will also go through the details of how receive is implemented in the BEAM in later chapters.

With the new message_queue_data flag introduced in Erlang 19 you can trade memory for execution time in a new way. If the receiving process is overloaded and holding on to the main lock, it might be a good strategy to use the off_heap allocation in order to let the sending process quickly dump the message in an m-buf.

If two processes have a nicely balanced producer consumer behavior where there is no real contention for the process lock then allocation directly on the receivers heap will be faster and use less memory.

If the receiver is backed up and is receiving more messages than it has time to handle, it might actually start using more memory as messages are copied to the heap, and migrated to the old heap. Since unseen messages are considered live, the heap will need to grow and use more memory.

In order to find out which allocation strategy is best for your system you will need to benchmark and measure the behavior. The first and easiest test to do is probably to change the default allocation strategy at the start of the system. The ERTS flag +hmqd sets the default strategy to either off_heap or on_heap. If you start Erlang without this flag the default will be on_heap. By setting up your benchmark so that Erlang is started with +hmqd off_heap you can test whether the system behaves better or worse if all processes use off heap allocation. Then you might want to find bottle neck processes and test switching allocation strategies for those processes only.

The first two bits are called the primary tag, and are used as follows:

An Immediate (tag 11) is a self-contained word, such as a small integer. It requires no additional words. The Boxed tag 10 is used for pointers to objects on the heap, except lists.

The Header tag 00 is only used on the heap for header words, which we will explain in Section 4.5. All objects on the heap use a whole number of machine words, generally starting with a header word. On the stack, 00 instead indicates a return address or Continuation Pointer — this is a practical choice which means no tagging/untagging of such pointers needs to be done, since return addresses will always be word-aligned anyway. This is a better tradeoff than using 00 for immediates, lists, or boxed values.

The separate List tag (01) is an optimization due to lists being so heavily used. This lets each cons cell be stored without a header word on the heap, saving 1/3 of the space otherwise needed for lists, as you will see in Section 4.3 and Section 4.4.

The Immediate tag is further divided like this:

Pids and ports are immediates and can be compared for equality efficiently. They are of course in reality just references, a pid is a process identifier and it points to a process. The process does not reside on the heap of any process but is handled by the PCB. A port works in much the same way.

There are two types of integers in ERTS, small integers and bignums. Small integers fit in one machine word minus four tag bits, i.e. in 28 or 60 bits for 32 and 64 bits system respectively. Bignums on the other hand can be as large as needed (only limited by the heap space) and are stored on the heap, as boxed objects (see Section 4.5).

By having all four tag bits as ones for small integers the emulator can make an efficient test when doing integer arithmetic to see if both arguments are immediates. (is_both_small(x,y) is defined as (x & y & 1111) == 1111).

The Immediate 2 tag is further divided like this:

Atoms are made up of an index in the atom table and the atom tag. Two atom immediates can be compared for equality by just comparing their immediate representation.

In the atom table atoms are stored as C structs like this:

Thanks to the len and the ord0 fields the order of two atoms can be compared efficiently as long as they don’t start with the same four letters.

The Catch immediate is only used on the stack. It contains an indirect pointer to the point in the code where execution should continue after an exception. More on this in Chapter 8.

The Nil tag is used for the empty list written [] in Erlang, called NIL in the C code. The rest of the word is filled with ones.