I was pro-systemd at the time of the controversial discussions. I still think it's a net positive relative to what was there before. But personally speaking, it's only the core of the software (service management) that improved things for me. The other things (timers, journald) I either ignore or don't like, but perhaps they're useful for distributions.
jauntywundrkind 2 hours ago [-]
I've been quite happy with -boot, -networkd, -resolved. -logind too, what a powerful upshift in routing input.
I forget what but timesync is missing a lot of nice to haves. But ditching grub for something much much much more manageable & a joy to use. Networkd and resolved have both had a lot of nice clear straightforward options, and I love how systemd unit files snap together for these networking concerns.
Journald has been ok for me, and I use it a lot, but not intensively. Having adequate performance is definitely a huge concern.
ahartmetz 2 hours ago [-]
Yes, systemd's weirdly bundled system services largely suck. The ones that I'm sure about: journald is more annoying to use than syslog (it takes forever - like a minute on a top of the line CPU - to scan two months worth of logs) and resolvd has or used to have basic bugs that other systems don't have.
msgodel 4 hours ago [-]
I think if it was just service management most people wouldn't mind it nearly as much.
haiku2077 5 hours ago [-]
I've worked as a sysadmin/devops for a double digit number of years.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
thyristan 5 hours ago [-]
And made a number of new ones appear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
idoubtit 4 hours ago [-]
The grand-parent post was about the general consensus: sysadmins generally like systemd. So I'm not sure answering individual concerns is useful, but I'll take the bait...
> now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng.
And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files.
> Boot and shutdown has previously been deterministic, now things just randomly hang.
That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
> Network device naming is also a fun one.
The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.
thyristan 2 minutes ago [-]
> No, you don't have to, because a syslog daemon is not required any more.
That is "required" for the single-computer-sysadmin. If all you know is your laptop, then journald might be fine. If your computers are cattle and have numbers, you don't want to read your logs on each individual computer, if possible you don't want to read logs at all, you want some automation to do it for you, aggregate it, warn you about things, etc. journald just makes that harder, and going through some syslog protocol implementation (of which there are a few, you are right there) is the only relevant game in town.
> IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
Dependencies and insserv were a later addition. Basically you just defined a boot order by numbering your /etc/rc[0-6].d/S* links in ascending order, either manually or in a dependency-based fashion through insserv. This means that you can have a suboptimal ordering if those dependencies are wrong or you choose your ascending order poorly. But it also means that once you have a working boot order, it can be repeated, and it will be repeated. That is what I mean by determinism. Once you have a working boot, it will keep working. Solaris SMF did something more modern, you would define dependencies in service definitions, and it would compute a static deterministic ordering from that, without those /etc/rc[0-6].d/S* symlinks. Deterministic, but dependency-based. If it has worked once, it will work again. And they even had a rollback: if after a change, the next boot wouldn't work, you could go back to the previous boot order and fix things from there. And neither had race conditions, because you usually wouldn't do things in parallel, one service would wait for the previous one in the boot order (parallel was possible but rarely used, because of the inherent fragility and possible race conditions).
Systemd does it dynamically on every boot, no precomputation, just looking at all the files and starting everything that could possibly be started at the same time, in parallel. The reason for this is that this makes Lennarts laptop boot faster, which is true in the ideal case. But it is neither deterministic, nor debuggable. And other than the previous methods, it can lead to race conditions.
Note that there is one more race condition at boot, which is races against kernel devices and systemd-udevd. That neither got worse nor better with systemd, it just lead to the kernel people giving up and handling more things like firmware loading in kernel.
> The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel.
The Linux kernel does not have a net.ifnames parameter, and the kernel interface naming scheme is still eth[0-9]+. When you stick that parameter in the kernel commandline, the kernel ignores it and only systemd and systemd-udevd interpret it later on.
Please, if you want to sing the praises of software or deflect blame from broken crap, make sure that you know what you are talking about...
msgodel 4 hours ago [-]
It's a little weird to reply to someone complaining about administrative problems caused by systemd with "generally sysadmins seem to like it."
That kind of response is exactly why everyone gets emotional in these discussions. Technical problems are one thing but when you gaslight people experiencing and reporting legitimate problems you should not be surprised when they get mad and rip your solution out.
GP is being far kinder to the systemd devs than they deserve, I gave up on the project years ago for exactly this reason.
mindslight 2 hours ago [-]
> consistent-net.rules nailed it to a mac address on first boot
fails even harder for
> put in a live-CD to debug something and your network devices are suddenly weird
I don't know that there is really any good answer here. Just methods that break in ways you're used to, and methods that break in ways you aren't. I appreciate the attempt at static naming based on hardware connection, even if the realities of how these things are presented by the bios make the results an inscrutable mess.
Maybe pure enx would have been better? But that looks like crap for even the simple cases, and fails if you swap out a piece of hardware or start doing weird things with MAC addresses.
thyristan 24 minutes ago [-]
The only consistent naming scheme I've ever seen working was Solaris on Sun hardware. But that only worked because they controlled both the hardware and the software side of things.
And yes, consistent-net.rules doesn't solve LiveCDs and reinstallations. But as I've said, systemd doesn't solve those either, and in most cases actively makes them worse.
vinceguidry 5 hours ago [-]
I used to be fine with it, until I started writing my own software tools, then looking at how to actually do stuff with systemd and I'm like, nope nope nope nope. At some point when I have space for a new project I'm going to migrate from Arch to Obarun, which uses a modern update on the excellent daemontools to provide init. When systemd works it's mostly fine, when it breaks, path of least resistance is often to reinstall the OS. Which is fine if you're just working on it. Painful if it's your personal machine, and those are what's most likely to break systemd.
akdev1l 4 hours ago [-]
> when it breaks, path of least resistance is often to reinstall the OS
Can you elaborate how do you even break this?
Adding a systemd unit cannot really break much on systemd itself. Unless you’re messing with internal systemd stuff, I am not sure what is even there to break.
jauntywundrkind 57 minutes ago [-]
This doesn't match my experience at all. For the life of me, I cannot imagine what you are talking about, what any of your issues are. Did you file any tickets, or report any issues?
What about "actually doing stuff with systems" was "nope nope nope nope"? What situations did systemd break in? What did you try before giving up to reinstall the OS?
Thise fud train is cast aspersions, no meat.
shrubble 5 hours ago [-]
My view is pragmatic: if you are managing the server and I am a user, then use systemd if you want to.
If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
hungryhobbit 5 hours ago [-]
FreeBSD is many great things, but ... a pragmatic choice?
shrubble 4 hours ago [-]
It works for many, including Netflix. The jails are better integrated as a lightweight container system than Docker is on Linux, IMHO.
linksnapzz 5 hours ago [-]
Yes; it does work nicely in many circumstances.
scraptor 5 hours ago [-]
I don't hate journald because it's not plaintext, I hate it because it's worse than plaintext.
Somehow journald manages to provide a database which is 40x slower to query than running grep on a compressed text file.
I'm all in favour of storing logs in an indexed structured format but journald ain't it.
bombela 4 hours ago [-]
Even doing zcat | grep is faster than journald.
I now turn off journald and use rotated pain text log files. It's more efficient in all metrics.
mrweasel 5 hours ago [-]
Journald is an odd one. I don't think it being a binary log/database makes sense. If you have a tiny operation, with a single server, then the binary database doesn't really make sense, having plain text is just easier and faster. If you're a bigger operation, you'll have a central logging solution, in which case you need journald to store the longs as plain text as well, before you can do log shipping.
The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
akdev1l 4 hours ago [-]
afaik journald can just forward logs via rsyslog directly to a remote server
Why would it need to store plaintext locally?
mrweasel 4 hours ago [-]
Doesn't that still involve a conversion? I believe that rsyslog can read the journald database, but you're typically not querying syslog data directly, so there's a conversion between rsyslog and logstash, Splunk, Datalog, whatever.
Does anyone actually use journald? The last time I tried(2 years ago?) it didn't even work with any log management software (like cloudwatch for example).
You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
nosioptar 5 hours ago [-]
I use journald whenever I feel my blood pressure getting too low.
It's slow, truncates lines, and doesn't work well at all with less. It's almost like Pottering created it so that pulseaudio wouldn't be his worst program anymore.
tylerjl 5 hours ago [-]
One convenience of journald is that it exposes a single place to plug in log collection for observability tooling
opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup
Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler
thyristan 5 hours ago [-]
That is what syslogd has been doing since forever. Journald actually made this harder due to not supporting the established syslog protocol.
gerdesj 5 hours ago [-]
Pipe it into lnav eg:
journalctl -b | lnav
or use -f instead of -b for follow instead of everything since boot. Now you have a colourised journal and the power of lnav.
stouset 5 hours ago [-]
I architected and built our entire log-ingestion pipeline for intrusion detection on it, at Square.
I built a small Ruby wrapper around the C API. Then I used that to slurp all the logs, periodically writing the current log ID to disk. Those logs went out onto a pubsub queue, where they were ingested into both BigQuery for long-term storage / querying, and into our alerting pipeline for real-time detection.
Thanks to journald, all the logs were structured and we were able to keep a bunch of trusted metadata like timestamp, PID, UID, the binary responsible, etc. (basically anything with an underscore prefix) separate from the log message all the way to BigQuery. No parsing, and you get free isolation of the trusted bits of metadata never intermingling with user-controlled attributes.
Compared to trying to durably tail a bunch of syslog files, or having a few SPOF syslog servers that everyone forwarded to, or implementing syslog plugins, this was basically the Promised Land for us. I think we went from idea to execution in maybe a month or two (I say “we” but really this was “me”) and rolled it out as a local daemon to the entire fleet of thousands. It has received—I think—one patch release in its six+ year lifetime, and still sits there quietly collecting everything to be shipped off-host.
The only issue we’ve ever really ran into that I never figured out is a handful of times per year (across a fleet of thousands) the journald database corrupted and you couldn’t resume collecting from the saved message ID. But we were also on an absolutely ancient version of RHEL, and I suspect anything newer probably fixed that bug. We just caught the error and restarted from an earlier timestamp. We built the whole thing around at-least-once delivery so having duplicates enter the pipeline didn’t really matter.
Damn, honestly at this point I’m wishing I’d pushed to open source it.
Ironically, actually, I did write a syslog server that also forwarded into this pipeline since we had network hardware we couldn’t install custom services onto but you could point them at syslog. I also wrote this in Ruby, using the new (at the time) Fibers (“real” concurrency) feature. The main thread fired up four background threads for listening (UDP, UDP/DTLS, TCP, TCP/TLS), and each of those would hand off clients to a dedicated per-connection worker thread for message parsing. Once parsed they went onto one more background thread for collecting and sending to PubSub. Even in Ruby it could handle gazillions of messages without breaking a sweat. Fun times.
Since I’m rambling, we also made cool use of zstd’s pre-trained dictionary feature. Log messages are small and very uniform so they were perfect for it. By pre-sharing a dictionary optimized for our specific data with both ends of the pubsub queue, we got something like 90%–95% compression rates. Given the many terabytes of logs we were schlepping from our datacenters to GCP, this was a pretty nice bit of savings.
inetknght 5 hours ago [-]
For debugging my desktop:
journalctl --follow --tail --no-trunc -b 0
For anything else: export to a syslog server, which basically any tool that matters will support in some fashion.
LargoLasskhyfv 2 hours ago [-]
journalctl: unrecognized option '--tail'
thyristan 5 hours ago [-]
Well, everyone and noone uses journald. The usual way to log is journald -> (r)syslogd -> remote log destination(s). I've never actually seen any other way.
The preferred journald way of "fetch your logs periodically" wouldn't pass any audit and doesn't work with any kind of log processing software that people use.
sethops1 5 hours ago [-]
I use it everyday?
NewJazz 5 hours ago [-]
Cloudwatch fucking sucks.
Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector)
Even ChromeOS uses it, even on devices that still use Upstart.
yjftsjthsd-h 58 minutes ago [-]
> Even ChromeOS uses it, even on devices that still use Upstart.
> Jounald is deprecated and is about to be removed.
bigstrat2003 4 hours ago [-]
You're being downvoted but you're absolutely right. The fact that Cloudwatch doesn't support journald is a major, major fail on AWS' part. It's not like this is new or obscure software.
NewJazz 2 hours ago [-]
I'm just surprised anyone wants to use cloudwatch when they don't need to. It is expensive, and far from the best observability platform.
spicyusername 7 hours ago [-]
I remember when I first upgraded a system to a version using it. This was probably a RHEL 6 -> 7 update back in 2014 or so.
It made so many common things I had to do so as a sysadmin super straightforward.
freedomben 6 hours ago [-]
Same, that was my first experience as well. It was a breath of fresh air in many ways, and after my initial learning of the new syntax, I quickly became a convert.
That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
Propelloni 5 hours ago [-]
And some things got super hard to do, e.g. sequence. I guess it is sorted now, but either way, I wouldn't want to go back.
rootnod3 6 hours ago [-]
I wouldn’t call it a success it all. Almost managed to do was moving or forcing other software to get locked in. GDM, Podman, Pipewire…just to name a few.
fsflover 3 hours ago [-]
As another comment says, it's a success in the same way Windows has been a success.
plantain 5 hours ago [-]
journald is such a disaster. How is it possible to be so much slower than even gzipped plain text.
artooro 5 hours ago [-]
I've been a fan of systemd even if just for how much simpler creating services is. I just need to write a simple config file instead of a complex init script.
haiku2077 5 hours ago [-]
+1 - there was a misconception that initscripts were easy or simple - if you wanted _correct, bug free_ initscript you were probably looking at around 100 lines of shell script.
SEJeff 5 hours ago [-]
Or if the service didn't support pam_limits because it was legacy trash, you had to hack something into the initscript like `ulimit -n XYZ` and restart it. Now things like this are trivial and easy to solve. Using systemd makes large scale Linux systems administration much easier.
Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in.
Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.
iambvk 6 hours ago [-]
I am not a sysadmin -- so I keep forgetting the command to see logs cause I only need it once every 3 months.
So, systemd change is a pain for me.
hearsathought 4 hours ago [-]
Keep a file with a list of commands used semi-regularly. Or have a script dump your command history into a file.
Better to spend a minute searching for a command in a file than to waste an hour recreating it.
(Not that I think journalctl -u even needs an alias tbh but if you want it you can have it)
slashdev 5 hours ago [-]
AI is so helpful for doing those kinds of tasks in the terminal that you only do a couple times a year and can never remember the exact incantation
hundchenkatze 5 hours ago [-]
You only need to look those things up once, put it in a snippets.txt file, and then search in the file when you need it.
dontlaugh 7 hours ago [-]
These were pretty much my feelings back when I envied launchd and was frustrated with some upstart behaviour. Systemd turned out better than both.
pram 5 hours ago [-]
I personally love the init/service/unit-file portion of systemd and have few complaints. It has a lot of powerful features for security, cron, etc that are very simple to use. I also really like journald.
The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
thyristan 5 hours ago [-]
And those ancillary managers are usually half-baked or outright broken and dangerous, in that they only implement half of the stuff they are trying to replace. E.g. https://github.com/systemd/systemd/issues/25676
bryanlarsen 5 hours ago [-]
Hard disagree. Name resolution feels like a solved problem, so why fix it if it isn't broken?
But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better.
I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved.
Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
msgodel 4 hours ago [-]
I think even for new people the systemd configs are much more complex (and the APIs are certainly much more complex) than the libc ones which worked well (often better.)
The problem is that almost nothing in Systemd shares theory with the rest of the OS. It's entirely its own island of new ideas, commands, and configuration language that you have to learn and memorize.
christophilus 4 hours ago [-]
I like systemd, but do agree that it's clunky in a lot of places. I'd be interested in hearing from anyone who has used Obarun or its 66suite.
LargoLasskhyfv 2 hours ago [-]
You can have S66 with AntiX too. Just use the 'Init Diversity Edition' (Respin), and pick it from several other options. AFAIK the author of S66 directly collaborated with Antix, to make it perfect.
pessimizer 5 hours ago [-]
No, it's terrible. I think what they mean by "success" here is that it functions, and is easier to use and more sturdy than init scripts. But any number of things would have been better than those. Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now.
Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
Analemma_ 5 hours ago [-]
> Instead of writing and adopting one or more, Linux allowed Redhat to take over the last piece of itself, its spine, after which Redhat sold itself to IBM. Linux is an IBM product.
Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
yjftsjthsd-h 51 minutes ago [-]
Those alternatives exist, actually. OpenRC, dinit, runit, and s6 are all good options and all used by some distros.
msgodel 4 hours ago [-]
OpenRC is great, there's just no flashy marketing behind it or team of consultants pushing it.
EDIT:
Heh. At least I can debug a shell script, I can't debug a config file.
I think this is a fundamental disagreement many people have in what's ultimately desirable when designing software. I personally can't stand non-trivial config files and both at work and at home have wasted way too much time pulling apart software consuming them.
The problem with systemd is that, because of all the new userspace APIs it adds, we can't just agree to disagree. You either target a Systemd OS and the users get to suffer non-trivial, often broken, configs or you target a normal GNU/Linux OS and the systemd users suffer all kinds of surprises when systemd rug pulls them.
akdev1l 4 hours ago [-]
#!/sbin/openrc-run
depend() {
need net
}
# Export some ENVs for it
export SERVER_ADDRESS="0.0.0.0:8088"
name="my-program"
description="My program"
command="/usr/local/bin/my-program"
# If it does not know how to background iself
command_background=true
pidfile="/run/${RC_SVCNAME}.pid"
# To have logs
output_log="/var/log/${RC_SVCNAME}.log"
error_log="/var/log/${RC_SVCNAME}.err"
It seems that it still uses Turing-complete shell scripts as configuration. Hard pass.
jmclnx 4 hours ago [-]
>Who exactly was going to write the systemd alternative?
And that quote proves 100% what is wrong with Linux these days.
In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more.
That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch.
Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then.
Seems these days we are dealing with some corporate behemoth drowning in red tape.
johnea 3 hours ago [-]
I'm sure it's a success for someone 8-/ Mostly IBM.
It's certainly been a successful takeover of all of the userspace runtime by one giant integrated set of executables.
Pwned off as an init replacement, then feature creeping it's way into every nook of userspace.
For me, it's still a disaster for distro and system s/w independence and the "do one thing well" philosophy.
While many welcome our new system overlords, the s/w is still not liked by many many people...
Move into the void! The DJB way and runit await...
tylerjl 5 hours ago [-]
Hi, author here. The title is intentionally a little attention-grabbing hyperbole but I've appreciated the discussion and feedback about the post. Thanks for reading!
rootnod3 4 hours ago [-]
This is a nit-pick, but if you picked the title that way to BE hyperbolic, I am not sure if that is the best thing to do for a technical discussion. It sets an internal opinion before even clicking on it. It subconsciously makes people go into the article with an already preconceived notion.
fsflover 5 hours ago [-]
> The unix philosophy cries out: is this the end of Linux (or, as many are calling it, GNU plus Linux)?
I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence:
> i used to think that systemd was made the default and adopted by most distros because of its ease of use and the fact it supplied a whole bunch of things in one suite and i see where the appeal is in that but after switching to artix openrc, im just lost on why they decided to use systemd when openrc is objectively better when it comes to being an init system and for managing services, and all the other components of systemd suite can just be replaced, like why would they do this?
I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on.
It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns.
... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it.
Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different.
I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems.
(Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)
pragmatic 2 hours ago [-]
Loonix users: We hate windows
Loonix developers: Now Loonix is windows
Loonix users: booo
… several hours later
Loonix users: yaaaay
/s
I remember running Linux servers for the first time and couldn’t believe there wasn’t a straight forward way to get something to run on startup. Ask glue and duct tape and non portable.
Don’t get me started on the Python situation.
I assume it’s much better now.
With the clown services U hadn’t had to run bare Linux in a long time.
jmclnx 5 hours ago [-]
systemd has been a success for 2 groups, Windows admins and Fortune 500 Companies.
For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though.
For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.
rootnod3 3 hours ago [-]
DRM is almost there. Try using Netflix on a *BSD, you can't. The DRM Netflix uses is not supported on BSDs.
up2isomorphism 6 hours ago [-]
There are numerous distributions not using systemd. So adding “complete, utter, unmitigated “ in front of success seems far-fetched.
freedomben 6 hours ago [-]
The whole post does have a somewhat hyperbolic tone to it. Personally, I agree with the content and analysis, but I could see the language could feel a bit off-putting, especially given the past flame wars over systemd
jhickok 6 hours ago [-]
It's a lighthearted post, so probably not worth tone policing.
freedomben 5 hours ago [-]
Agreed, I didn't mean to tone police, just acknowledging it might be off-putting to some people who got wrapped up in the battles back in the day. I wouldn't suggest the author make any changes
I forget what but timesync is missing a lot of nice to haves. But ditching grub for something much much much more manageable & a joy to use. Networkd and resolved have both had a lot of nice clear straightforward options, and I love how systemd unit files snap together for these networking concerns.
Journald has been ok for me, and I use it a lot, but not intensively. Having adequate performance is definitely a huge concern.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
> now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng.
And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files.
> Boot and shutdown has previously been deterministic, now things just randomly hang.
That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
> Network device naming is also a fun one.
The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.
That is "required" for the single-computer-sysadmin. If all you know is your laptop, then journald might be fine. If your computers are cattle and have numbers, you don't want to read your logs on each individual computer, if possible you don't want to read logs at all, you want some automation to do it for you, aggregate it, warn you about things, etc. journald just makes that harder, and going through some syslog protocol implementation (of which there are a few, you are right there) is the only relevant game in town.
> IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
Dependencies and insserv were a later addition. Basically you just defined a boot order by numbering your /etc/rc[0-6].d/S* links in ascending order, either manually or in a dependency-based fashion through insserv. This means that you can have a suboptimal ordering if those dependencies are wrong or you choose your ascending order poorly. But it also means that once you have a working boot order, it can be repeated, and it will be repeated. That is what I mean by determinism. Once you have a working boot, it will keep working. Solaris SMF did something more modern, you would define dependencies in service definitions, and it would compute a static deterministic ordering from that, without those /etc/rc[0-6].d/S* symlinks. Deterministic, but dependency-based. If it has worked once, it will work again. And they even had a rollback: if after a change, the next boot wouldn't work, you could go back to the previous boot order and fix things from there. And neither had race conditions, because you usually wouldn't do things in parallel, one service would wait for the previous one in the boot order (parallel was possible but rarely used, because of the inherent fragility and possible race conditions).
Systemd does it dynamically on every boot, no precomputation, just looking at all the files and starting everything that could possibly be started at the same time, in parallel. The reason for this is that this makes Lennarts laptop boot faster, which is true in the ideal case. But it is neither deterministic, nor debuggable. And other than the previous methods, it can lead to race conditions.
Note that there is one more race condition at boot, which is races against kernel devices and systemd-udevd. That neither got worse nor better with systemd, it just lead to the kernel people giving up and handling more things like firmware loading in kernel.
> The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel.
Utterly totally completely wrong. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
The Linux kernel does not have a net.ifnames parameter, and the kernel interface naming scheme is still eth[0-9]+. When you stick that parameter in the kernel commandline, the kernel ignores it and only systemd and systemd-udevd interpret it later on.
Please, if you want to sing the praises of software or deflect blame from broken crap, make sure that you know what you are talking about...
That kind of response is exactly why everyone gets emotional in these discussions. Technical problems are one thing but when you gaslight people experiencing and reporting legitimate problems you should not be surprised when they get mad and rip your solution out.
GP is being far kinder to the systemd devs than they deserve, I gave up on the project years ago for exactly this reason.
fails even harder for
> put in a live-CD to debug something and your network devices are suddenly weird
I don't know that there is really any good answer here. Just methods that break in ways you're used to, and methods that break in ways you aren't. I appreciate the attempt at static naming based on hardware connection, even if the realities of how these things are presented by the bios make the results an inscrutable mess.
Maybe pure enx would have been better? But that looks like crap for even the simple cases, and fails if you swap out a piece of hardware or start doing weird things with MAC addresses.
And yes, consistent-net.rules doesn't solve LiveCDs and reinstallations. But as I've said, systemd doesn't solve those either, and in most cases actively makes them worse.
Can you elaborate how do you even break this?
Adding a systemd unit cannot really break much on systemd itself. Unless you’re messing with internal systemd stuff, I am not sure what is even there to break.
What about "actually doing stuff with systems" was "nope nope nope nope"? What situations did systemd break in? What did you try before giving up to reinstall the OS?
Thise fud train is cast aspersions, no meat.
If it’s my responsibility then I will use a non-systemd distribution or if I have a choice, FreeBSD.
I now turn off journald and use rotated pain text log files. It's more efficient in all metrics.
The only use case where the binary format might make sense is if you ship journald logs to another central journald instance. That's just very much an edge case.
Why would it need to store plaintext locally?
https://www.theregister.com/2017/07/28/black_hat_pwnie_award...
You had to either use some (often abandoned) third party tool or defeat the purpose by just reconfiguring everything to dump a text log to a file.
It's slow, truncates lines, and doesn't work well at all with less. It's almost like Pottering created it so that pulseaudio wouldn't be his worst program anymore.
opentelemetry-collector, promtail, and so on have native plugins for it, which makes aggregation easier to setup
Most tools have "tail this plaintext file" as well, but if it's all flowing to journald, setting up log collection ends up being that much simpler
journalctl -b | lnav
or use -f instead of -b for follow instead of everything since boot. Now you have a colourised journal and the power of lnav.
I built a small Ruby wrapper around the C API. Then I used that to slurp all the logs, periodically writing the current log ID to disk. Those logs went out onto a pubsub queue, where they were ingested into both BigQuery for long-term storage / querying, and into our alerting pipeline for real-time detection.
Thanks to journald, all the logs were structured and we were able to keep a bunch of trusted metadata like timestamp, PID, UID, the binary responsible, etc. (basically anything with an underscore prefix) separate from the log message all the way to BigQuery. No parsing, and you get free isolation of the trusted bits of metadata never intermingling with user-controlled attributes.
Compared to trying to durably tail a bunch of syslog files, or having a few SPOF syslog servers that everyone forwarded to, or implementing syslog plugins, this was basically the Promised Land for us. I think we went from idea to execution in maybe a month or two (I say “we” but really this was “me”) and rolled it out as a local daemon to the entire fleet of thousands. It has received—I think—one patch release in its six+ year lifetime, and still sits there quietly collecting everything to be shipped off-host.
The only issue we’ve ever really ran into that I never figured out is a handful of times per year (across a fleet of thousands) the journald database corrupted and you couldn’t resume collecting from the saved message ID. But we were also on an absolutely ancient version of RHEL, and I suspect anything newer probably fixed that bug. We just caught the error and restarted from an earlier timestamp. We built the whole thing around at-least-once delivery so having duplicates enter the pipeline didn’t really matter.
Damn, honestly at this point I’m wishing I’d pushed to open source it.
Ironically, actually, I did write a syslog server that also forwarded into this pipeline since we had network hardware we couldn’t install custom services onto but you could point them at syslog. I also wrote this in Ruby, using the new (at the time) Fibers (“real” concurrency) feature. The main thread fired up four background threads for listening (UDP, UDP/DTLS, TCP, TCP/TLS), and each of those would hand off clients to a dedicated per-connection worker thread for message parsing. Once parsed they went onto one more background thread for collecting and sending to PubSub. Even in Ruby it could handle gazillions of messages without breaking a sweat. Fun times.
Since I’m rambling, we also made cool use of zstd’s pre-trained dictionary feature. Log messages are small and very uniform so they were perfect for it. By pre-sharing a dictionary optimized for our specific data with both ends of the pubsub queue, we got something like 90%–95% compression rates. Given the many terabytes of logs we were schlepping from our datacenters to GCP, this was a pretty nice bit of savings.
The preferred journald way of "fetch your logs periodically" wouldn't pass any audit and doesn't work with any kind of log processing software that people use.
Plenty of log shippers can slurp journald. (Fluentd, filebeat, vector)
Even ChromeOS uses it, even on devices that still use Upstart.
I was curious about this, because I thought journald hard required systemd as pid 1, so I did a search, which promptly turned up https://www.chromium.org/chromium-os/developer-library/refer... -
> Jounald is deprecated and is about to be removed.
It made so many common things I had to do so as a sysadmin super straightforward.
That said, I have had some concerns over the years about the growing scope of systemd. I think history has shown my concerns were overblown, though I think it is definitely possible and even likely that people were affected by those arguments and maybe moderated the ambitions a little bit. That could be for better or worse, I don't know, but I do think it was a factor
Now it has gone a bit overboard. Some of the stuff like the dns resolver or the nspawn capability seem a bit over the top, but overall, it has massively improved all Linux distributions it is used in.
Never again will I worry about trash buggy init scripts not actually stopping a service due to a stale pid file. Now it puts the service into a control group and can kill all things in the control group even if the service is bad code.
So, systemd change is a pain for me.
Better to spend a minute searching for a command in a file than to waste an hour recreating it.
The parts I don’t really like are the half baked “ancillary” managers like resolvd, which are frankly baffling to use and tend to make simple things needlessly difficult.
But that's just like the old init & logging systems. They mostly worked, and yes it was annoying to relearn a new way of doing things. But the new way is better.
I've got software that works with resolvconf, NetworkManager and resolved. I definitely prefer resolved.
Yes, the cost of switching likely exceeds the benefits for you, and likely for many greybeards. But for those in complicated situations, and for the next generation of sysadmins who never have to learn the old way, resolved is nice.
The problem is that almost nothing in Systemd shares theory with the rest of the OS. It's entirely its own island of new ideas, commands, and configuration language that you have to learn and memorize.
They basically mediate all Linux access to the hardware, and can add arbitrary dependencies to the system at will. They could decide that we're all going to use leftpad now.
Cutting off my hand to get rid of that cancerous mole might have saved my life, but I wish you had just cut off the mole.
Who exactly was going to write the systemd alternative? If by "Linux" you mean the kernel devs, that was never going to happen-- systemd is a middleware stack living in userspace, the kernel guys were never going to get involved. If you mean some distro, other distros did propose some alternatives to systemd back in the day (e.g. upstart), and they all eventually abandoned them and switched to systemd instead, because systemd was better.
EDIT: Heh. At least I can debug a shell script, I can't debug a config file.
I think this is a fundamental disagreement many people have in what's ultimately desirable when designing software. I personally can't stand non-trivial config files and both at work and at home have wasted way too much time pulling apart software consuming them.
The problem with systemd is that, because of all the new userspace APIs it adds, we can't just agree to disagree. You either target a Systemd OS and the users get to suffer non-trivial, often broken, configs or you target a normal GNU/Linux OS and the systemd users suffer all kinds of surprises when systemd rug pulls them.
depend() { need net }
# Export some ENVs for it export SERVER_ADDRESS="0.0.0.0:8088"
name="my-program" description="My program" command="/usr/local/bin/my-program" # If it does not know how to background iself command_background=true pidfile="/run/${RC_SVCNAME}.pid" # To have logs output_log="/var/log/${RC_SVCNAME}.log" error_log="/var/log/${RC_SVCNAME}.err"
It seems that it still uses Turing-complete shell scripts as configuration. Hard pass.
And that quote proves 100% what is wrong with Linux these days.
In the 90s, if someone posted in USENET they want init to do X, at least 2 different solutions would appear with in a week, probably many more.
That happened to me when I went from Coherent OS to Slackware in the early 90s. I asked if there was a way to have virtual consoles on my VGA monitor and my B&W Hercules monitor (386sx), or to have at least 1 VC on the Hercules. Coherent 486 supported that, Linux did not. A post appeared in a couple of days with a patch.
Now, between the Linux Foundation and the Companies that owns it, that would never happen. People who have gotten into Linux in the last 20 years cannot imagine how responsive the developers were to user requests back then.
Seems these days we are dealing with some corporate behemoth drowning in red tape.
It's certainly been a successful takeover of all of the userspace runtime by one giant integrated set of executables.
Pwned off as an init replacement, then feature creeping it's way into every nook of userspace.
For me, it's still a disaster for distro and system s/w independence and the "do one thing well" philosophy.
While many welcome our new system overlords, the s/w is still not liked by many many people...
Move into the void! The DJB way and runit await...
I still see no counterarguments here. Meanwhile, systemd, like cancer, is eating Linux from the inside, breaking its flexibility and therefore future resilence:
https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd
https://news.ycombinator.com/item?id=42918448
https://news.ycombinator.com/item?id=42889792
(I reposted my comment from previous submission)
I was following where the author was coming from until they quoted this, and rather than addressing it they just slammed it, slammed the author, and moved on.
It's weird to say that ini files are not a domain specific language. And this kind of hand waving away complexity is the crux of systemd putting a bad taste in my mouth. The whole thing is a vast kingdom of nouns (cf Yegge). Forgoing expressing semantics with the config file syntax means that expressing those semantics requires creating even more nouns. Splaying the config across a bunch of tiny files forgoes juxaposition and makes for even more nouns. Owning the aspect of whether a given config file is enabled or not creates even more nouns. Needing tools to analyze the distilled config (eg dependency graph) necessitates even more nouns.
... and this was all dumped on a community that very much focuses on a smaller number of verbs - ie command line utilities. The main problem with SysV scripts is that they are bare scripts doing anything they want with those longstanding verbs. So yes, some abstraction was necessary, and that was inevitably going to seem a bit unnecessary-complexity. But systemd just took that dynamic and went overboard with it.
Sure, I can read "type=forking" and know what that means, but no, I cannot change it without looking up the documentation. Any interaction with systemd beyond start/stop/journalctl involves looking up the documentation, re-figuring-out its implicit logical model, copying the right magical incantation, and then after I've solved the problem promptly forgetting everything I figured out because the next time I touch systemd I'm guaranteed to need to know something different.
I don't dislike it enough to desperately want to move to a different init system (although if there were an easy alternative on NixOS I'd try), and I'm certainly not going to defend SysV or BSD init scripts. But it certainly feels like there is plenty of room to implement systemd's comprehensive functionality in a more user-friendly discoverable way, especially for the main type of advanced-but-casual user that interacts with init systems.
(Also as a fellow NixOS user I think it's easy to have a blindspot about systemd because its config files align with our config format. A more user-friendly concise config format would make NixOS have to do more work to splay it out for its own composability)
Loonix developers: Now Loonix is windows
Loonix users: booo
… several hours later
Loonix users: yaaaay
/s
I remember running Linux servers for the first time and couldn’t believe there wasn’t a straight forward way to get something to run on startup. Ask glue and duct tape and non portable.
Don’t get me started on the Python situation.
I assume it’s much better now.
With the clown services U hadn’t had to run bare Linux in a long time.
For UN*X people, all it does is move Linux closer to emulating Microsoft Windows. It has left the UN*X philosophy behind. With secure boot and once Wayland becomes a real thing, the last step is in place. All that will be needed to become a M/S Windows Clone is geo-location and full DRM. That means you will only be able to take screen prints of "approved" applications. I heard SUSE may enable a kind of geo-location, not sure if I read that article correctly though.
For now, at least there are the *BSDs, they allow me to make a PC work the way I want it to, not a way Fortune 500 Companies dictate it should work.