Many of the ideas inside the library are also explained in the Structured Concurrency blog post on the authors homepage: http://250bpm.com/blog:71
I really sympathize with a lot of them, e.g. scoping the lifetime of all child processes (goroutines, ...) to the lifetime of their parent and having good cancellation/timeout/close mechanisms for everything. I'm trying to apply these patterns in Go and it can solve a lot of scenarios in a more obvious and less spaghetti fashion.
Libdill seems to bringt a view more primitives to the table which for example make cancellation and waiting for child processes to finish easier. But the basic mechanism can be also applied in Go (e.g. via context.Context and a WaitGroup/Channel for waiting on the childprocess to finish) and therefore most likely also to libmill.
However, there are couple of simplifications that were done in the meantime.
Most importantly, the article goes into great pains to define a single API that deals with both soft and hard cancellation of coroutines.
The library takes advantage of the fact that soft cancellation is always application-specific and provides a single API to do hard cancellation (hclose(h)). User is free to implement the soft cancellation themselves.
First, since people generally assume replies are disagreement, yes, goroutines can not be externally cancelled and I don't expect that to ever change.
However, you can often mostly simulate what this library does in Go fairly effectively, since the goroutines end up stereotypically used. For instance, if your goroutine's purpose in life is to read from a socket, you can close the socket from another goroutine and that will result in all reads immediately failing. If you close the resources the goroutine expects, you get similar results.
You do have to structure your code for this, though; if the goroutine that you're trying to close both opens a socket and listens for it, then another goroutine trying to close that socket may fail if it tries before the first one even has the socket open. One solution I've sometimes used is to lock around a struct with a bool and a socket reference; the goroutine opens the socket, takes the lock, closes if the bool is true and updates the socket reference if it isn't to point to the socket, then drops the lock. (One lock is generally not a great penalty next to the costs of acquiring the socket.) If an external goroutine tries to close the resource, it takes the lock, closes the socket reference if available, and sets the "closed" bool if it isn't. It sounds complicated in English but isn't that much code.
It's a cute trick in C to turn all blocking calls into ECANCELLED results (no sarcasm, it is a cute trick), but generally goroutines should be small enough that they have an obvious and small set of "resources" that can be closed, where one such resource is definitely the dominant case. (And Go is all about considering the dominant case and not inserting tons of language features for the longer-tail cases.)
A "done" channel is a good solution if your goroutine already has a "select" statement in it. I've found that correct use of the select statement actually requires more such "structural" channels than is sometimes advertised; in particular, whenever you write code where you want the receiver to close a channel for some reason, the correct solution under current Go constraints is actually to add a channel that goes back to the sender, which the sender must be modified to check, and close that. I find rather more of my channels end up being "struct{}" than I expected at first.
To me, the trick with closing the resource sounds like a hack.
Imagine you wanted to generalise it: We have a goroutine that may use some resources: sockets, files, keyboard, screen etc. To shut it down you cut it off from all those resources and hope that it will die.
I'd agree it was a hack if it were ill-defined, but I don't think it is. I don't think Go introduced that trick. In many cases you have to write your code to be able to deal with the resource being unavailable at any moment anyhow, especially networks, where Go is on its home turf. (As I've said more than once, the farther you are from writing something that looks like a network server, the less happy you will be with Go.)
Yes, that's what I meant with context.Context (which encapsulates a cancellation channel) - but you can of course also use a plain channel. The difference is that Go is more explicit in this (you need a bigger amount of boilerplate code to setup cancellation and to wait for child processes to finish) and that you need to select on channels for reacting to cancellation signals instead of only relying on return values.
All in all I think you can achieve the goal with both approaches. Go favors to provide more basic building blocks whereas libdill seems to be a little bit more opinionated how your concurrency should be structured.
In the same vein, but with less idiomatic C, note also https://github.com/hnsl/librcd. I had the opportunity to work with it two years ago at Jumpstarter, and it was quite wonderful.
It's C-idiomatic. Whereas libmill takes Go's concurrency API a implements them in almost 1:1 manner in C, libdill tries to provide the same functionality via more C-like and POSIX-y API.
For example, choose is a function rather than a language construct, Go-style panic is replaced by returning an error code and so on.
Coroutines and processes can be canceled. This creates a foundation for "structured concurrency".
chdone causes blocked recv on the channel to return EPIPE error rather than a value.
chdone will signal senders to the channel as well as receivers. This allows for scenarios like multiple senders and single receiver communicating via single channel. The receiver can let the senders know that it's terminating via chdone.
libmill's fdwait was replaced by fdin and fdout. The idea is that if we want data to flow via the connection in both directions in parallel we should use two coroutines rather than single one.
There's no networking library in libdill. This is meant to be a separate project.
libmill was a project to copy Go's concurrency model to C 1:1 without introducing any innovations or experiments. The project is finished now. It will be maintained but won't change in the future.
libdill is a follow-up project that diverges from the Go model and experiments with structured concurrency. It is not stable yet and it may change a bit in the future.
libdill's development is much more recent: started january 11 (6 months old), 384 commits.
libmill is "Go concurrency in C" library and is significantly older: end of April 2014 (2 years old), 992 commits.
Looks like a very nice API. Now I'm waiting for an excuse to use it!
The only minor blemish I see is the need to call `fdclean()` before closing a file-descriptor. It introduces a slight non-orthogonality between libdill and straight posix.
Another approach might have been to wrap the fd up as a channel -- perhaps not one that transfers actual bytes, but which does signal availability.
The user would still need to to call chdone to clean up before calling fclose. But that would be a natural usage of the interface rather than a special-purpose wrinkle.
I am not really objecting to the need to clean up the fd-bookeeping data, just the interface.
And it's not much of an objection. I guess the intention behind libdill is that applications will do their I/O using some abstraction layer anyway -- and layer can be libdill aware enough to call fdclean().
If that is the design philosophy, then does that means there would be problems if someone used some big external libray that did lots of its own I/O inside a libdill coroutine?
Here's the use case that drives the design of fdclean: Imagine a third party library makes a callback to your code and hands you a file descriptor. You can use it temporarily, but you have no idea what will happen to it after you return from the callback function. Will it live on? Will it be closed? The only way to deal with that is to purge any cached info about the fd before returning and pretend that you never had anything to do with that file descriptor.
> What that means is that one coroutine has to explicitly yield control of the CPU to allow a different coroutine to run. In the typical case this is done transparently to the user:
And this is exactly what's wrong with coroutines. It's not "transparent" to the user, but actually implicit to the user and unpredictable. With that user has no control over switch between coroutines and no idea when it's happening, which forces him to think about synchronization and generally have a very bad time if the essential complexity of the task he needs to implement is not something as trivial, as talking to a database.
In contrast, having a keyword, like await, to let the user keep control of the switching doesn't have that problem and user doesn't have to think about synchronization. But such things cannot be called coroutines.
So, be wary when someone is trying to push coroutines onto you. It's never a good idea to have them.
It seems libdill supports channels -- blocking FIFOs.
In Golang (which seems to have indirectly inspired libdill), you would implement such logic by returning a channel (future) over which is sent just one value. That result itself might be a channel and so on.
No doubt the syntax in C would not be as convenient as in some other language -- but lightweight channels that work as first-class objects are a lot like futures.
One thing that I would love to see is the ability to schedule coroutines accross threads. Currently, you have to choose to spawn a new thread for your coroutine or keep it on the same thread. Although I believe adding a thread a pool and an assorted dispatch capability shouldn't be too hard.
I'm far out of my area of expertise here, but can anyone offer thoughts about the potential of wrapping these C routines and accessing them from python (Cython) to give python this level of concurrency?
I really sympathize with a lot of them, e.g. scoping the lifetime of all child processes (goroutines, ...) to the lifetime of their parent and having good cancellation/timeout/close mechanisms for everything. I'm trying to apply these patterns in Go and it can solve a lot of scenarios in a more obvious and less spaghetti fashion.
Libdill seems to bringt a view more primitives to the table which for example make cancellation and waiting for child processes to finish easier. But the basic mechanism can be also applied in Go (e.g. via context.Context and a WaitGroup/Channel for waiting on the childprocess to finish) and therefore most likely also to libmill.