> Prior to the process metadata caching, you could cause journald to go absolutely nuts querying the sending processes metadata from /proc by sending it messages full of newlines. Every newline-separated field is treated as a separate message with its own metadata to be acquired. The routines for fetching all that stuff from /proc granularly allocated and freed the data via libc as well, it was very painful.

This seems more like an indictment of journald's architecture than an argument against the cost of mallocs per-se

My point was that a malloc vs alloca in a full blown IO path, where you have sockets and posibly disk fsyncs is not all that significant, if at all. I understand that journald cares about performance. Micro-optimizations like malloc vs alloca are the least of your problems. Batching/buffering messages, an asyncronous model, plain O_APPEND, different files for logs/metadata etc like others and you have said ITT is what will matter. You can even use a sane dynamic string library with preallocation if you care that much. I haven't looked at the src, to be honest, but am I wrong?

journald does all its journal writing via mmap. When the window to write into is already mapped (it caches the mappings), there's relatively little per-message syscall overhead required.

Prior to process metadata caching, the per-message CPU cost was dominated by acquiring the sender process metadata from /proc, which was dominated by syscalls but the malloc()+free() overhead of all that junk was not insignificant.

I haven't done any journald profiling or hacking since those process metadata caching changes landed, but I imagine it's brought the per-message syscall overhead down to little more than pulling the message off the socket.

When I was experimenting with adding metadata caching, once caching could amortize the /proc-accessing syscall overhead across many messages the major pain was all the allocator activity in moving that metadata around and composing the KEY=VALUE clauses from that metadata. It was a death from a thousand cuts kind of situation, happening for every message.

The stack is the convenient place to efficiently do much of this granular, ephemeral stuff. But it's obviously inappropriate for allocations of user-controlled/unbounded size.

Thanks for the interesting and detailed info.

I mean the cost of hitting the disk/socket. The actual IO. There is also the syscall overhead but that doesn't matter /that much/ with IO usually, in comparation with the actual IO (unless, again, you are doing something very wrong - very small buffers etc ...). This of course depends on the deatils. And yeah, if it uses mmap then that syscall problem is no more. Problem I was refering was actual IO cost per message (fsync/msync per message?) to mantain consistency instead of caching for example or batching and grouping IO at the expense of consistency guarantees (maybe) or coherent data re. messages in realtime.

I agree that the stack is usefull and I see your point. But I would almost never resort to alloca. That is not a solution IMO. If you have that kind of costs with the joining of keys there are other solutions, I think. What about a dynamic string library, that over allocates (maybe a big chung on the first time in the 'new message' path) and then just uses the extra memory until you run out. Classic (and silly) strategy of allocating like n << 4 bytes every time you run out of space. This way you really just have a few allocations instead of hundreds maybe making it a non-issue (at the cost of prob. unused extra memory). Not as efficent as alloca but very close.

I you want to get fancy you can have efficent malloc pools to. But I think a semi-clever dynamic string lib would be able to handle the problem best.

I do think we'd be better off with the hot path being a simple append to a read-only file.

All this database-like junk should have been done asynchronously and persisted in ancillary read-write files kept adjacent to the logs.

Features like `systemctl status` that want to show the most recent log entries for a given service could have simply indicated 'potentially stale' when the logs were newer than the ancillary files, and support blocking on a refresh if desired.

I'm not a fan of how it is done today, and I've hacked a bit on journald. It doesn't belong in the critical path. There's a lot of that in the systemd project in general; tightly coupling complexity where it really should be loosely coupled at best. But I believe a lot of that is to be expected from a project aspiring to do too much with too little resources, shortcuts are taken regularly. Everyone experienced knows tightly coupling components eases the development of them.

my thoughts exactly. just because people are filing performance tickets doesn't mean that its performance bound application or framework. it could also, like illustrated many times before by people, just be bloated and convoluted code causing slowness.

To take such a position is to not think charitably about the intentions and competency of the developers, which I find personally absurd.

If you're not going to take the time to scrutinize the code and understand the problems, spreading uninformed FUD like a troll isn't helpful.

I have spent significant time with the journald code in the past. I understand its inner-workings and what performance bottlenecks it has suffered from.

Bloated and convoluted code is not how I would describe the causes. The systemd project's code is generally rather spartan, easy to read, simple C code.

Journald attempts to associate myriad process metadata with the logs it receives. This information is useful to the log consumers, and is actually somewhat impossible to reliably collect in lockstep with the messages being received, using today's kernel interfaces.

It will always be a racy endeavor to do this from userspace in the form of sampling /proc after the messages have been written and languished in a socket buffer for some non-deterministic amount of time.

What journald attempts to do is a best-effort approach of acquiring and associating this information with the messages in-line with the logging, to get it as close as possible. I believe at some point, if we continue using this kind of metadata-rich logging in Linux, that the kernel will grow new interfaces to reliably deliver this information in a socket sidechannel, much like how minimal sender credentials may be retrieved on UNIX domain sockets today.

Until then, journald will continue to find itself in the awkward position of being a kinda-sorta logging database which must sample piles of process metadata via /proc to describe the sender in what's appended to the journal.

But this is a necessary phase of progress we must pass through. Upstream kernel developers generally refuse to add new interfaces for this kind of thing until there's an established use case demanding it. So what we're all using in journald today is arguably an MVP, establishing that there's a market need for this level of information in our logs, which can then be used to compel upstream to help us make it both efficient and 100% reliable.

The process metadata caching that was added to improve the performance arguably traded accuracy of the metadata to do so. (and, one could argue, increased bloat and complexity) But in lieu of better kernel interfaces, it's the only choice other than giving up on logging the metadata entirely. The impression I got while working on the journald code pre-metadata-caching, was that the authors had assumed the kernel would evolve to make the information available efficiently. The kdbus debacle speaks to that trajectory, and its rejection from landing upstream definitely threw a wrench in the overall plan, I'm not sure the systemd project has ever fully recovered from that setback.

So please, refrain from making such uninformed speculative comments. I encourage you to instead get involved and help improve things where you can, if you care about running modern Linux systems.

journald seems to be overall a very bad idea.