Aloso's blog | Aloso's Blog

Template strings in Rust

Mon, 15 Dec 2025 00:00:00 GMT

Template strings have become widespread in modern programming languages, but Rust is a notable exception. Here I want to shed light on the design space and rationale for template strings in Rust, and present a proposal.

Motivation

Template strings allow interpolating values in a string. Some languages also support custom template strings for specialized use cases:

// template string in JavaScript
`Hello, ${person.name}!`

// tagged template string
sql`SELECT * FROM users WHERE id = ${handle} ORDER BY ${sortField};`
// this is equivalent to...
sql(
    ["SELECT * FROM users WHERE id = ", " ORDER BY ", ";"],
    handle, sortField
)

String interpolation is supported in JavaScript, C#, Python, Swift, Ruby, Kotlin, Scala and Dart, among others. Java and C++ are working on proposals to add string interpolation as well.

Strictly speaking, string interpolation isn’t required, especially in a language with a powerful macro system. Rust uses format_args! and related macros for string interpolation:

println!("x is {x}");   // Display impl
println!("x is {x:?}"); // Debug impl

But this approach has several disadvantages:

Flexibility: Rust doesn’t allow arbitrary expressions in format_args!, only single variables. We could allow more complex expressions here, but some things (e.g. nested string literals) would be difficult to support.
Complexity: The format_args! macro has many configuration options, making its bespoke syntax difficult to understand and remember. For example, here are some valid formatting arguments:
```
"{:#?} {:04} {:e} {:p} {:width$} {:<5} {:+5} {1}
{:[fill]>5} {:x} {:b} {:+} {:.7} {:.prec$} {:.*}"
```
These are all fairly simple, but you can also combine them (e.g. "{: >#w$.2?}"), and you need to be careful to write the options in the right order: fill, alignment, sign, alternate flag, leading zero, width, precision, formatting trait.
Syntax highlighting: Editors like VS Code provide syntax highlighting by special-casing format_args!, but most websites do not.
Extensibility: The built-in formatting options are powerful, but not extensible. For example, they don’t allow locale-aware formatting.
Availability: The formatting machinery is not available in no_std environments, e.g. Rust for Linux.
Performance: format_args! is not a zero-cost abstraction. The ufmt crate, for example, is faster and has a smaller binary size footprint.

Why shouldn’t we do this?

The RFC for implicit format args explains why it only allows single identifiers in format strings:

If any expressions beyond identifiers become accepted in format strings, then the RFC author expects that users will inevitably ask “why is my particular expression not accepted?”. This could lead to feature creep, and before long perhaps the following might become valid Rust:
println!("hello { if self.foo { &self.person } else { &self.other_person } }");
This no longer seems easily readable to the RFC author.

This is the strongest argument against expressions in template strings. This sentiment is also expressed in the most upvoted comment in the IRLO thread about the topic. However, this argument is flawed: It is a classic slippery slope fallacy.

This line of reasoning does not refute the argument that an if/else in a string literal can be unreadable. But the argument has a more serious flaw: It assumes that syntax must be chosen in a way that unreadable code becomes impossible.

Let’s consider this position for a moment! Rust, like all powerful languages, already allows for code that is deeply confusing:

fn punch_card() -> impl std::fmt::Debug {
    ..=..=.. ..    .. .. .. ..    .. .. .. ..    .. .. .. ..
    ..=.. ..=..    .. .. .. ..    .. .. .. ..    .. ..=.. ..
    ..=.. ..=..    ..=.. ..=..    .. ..=..=..    ..=..=..=..
    ..=..=.. ..    ..=.. ..=..    ..=.. .. ..    .. ..=.. ..
    ..=.. ..=..    ..=.. ..=..    .. ..=.. ..    .. ..=.. ..
    ..=.. ..=..    ..=.. ..=..    .. .. ..=..    .. ..=.. ..
    ..=.. ..=..    .. ..=..=..    ..=..=.. ..    .. ..=..=..
}

fn r#match() {
    let val: () = match match match match match () {
        () => ()
    } {
        () => ()
    } {
        () => ()
    } {
        () => ()
    } {
        () => ()
    };
    assert_eq!(val, ());
}

If we posit that unreadable code should be impossible, then the above should be rejected by the compiler. We could do this by disallowing ranges within ranges, and match constructs in a match scrutinee. But adding exceptions like these makes the grammar more and more complicated. Furthermore, no matter how many things we make illegal, there will always be potential for unreadable code.

Of course, the examples above are not real-world code, they are edge cases from the compiler’s test suite. But producing unreadable code is much simpler: You can write functions with 10 or more arguments. You can write loops within loops within loops. You can use unintelligible variable names. Etc.

The real solution is code review. In the TypeScript codebases where I have worked, with both experts and Junior devs, we have never had a problem with unreadable template strings.

The idea

Since Rust already has prefixed literals (b"", r"", c""), it’s intuitive to add another prefix, f:

// old
println!("{a} + {b} = {}", a + b);
buf.write_fmt(format_args!("{a} + {b} = {}", a + b))?;

// new
println(f"{a} + {b} = {a + b}");
buf.write_fmt(f"{a} + {b} = {a + b}")?;

This is possible since arbitrary string prefixes were reserved in the 2021 edition. Note that the println! macro can be replaced with a println() function, which accepts a single argument.

Formatting options are important, but the syntax can be greatly simplified using extension traits:

// old
println!("{foo:width$.2}");
// new
println(f"{foo.width(width).precision(2)}");

Not only is this easier to understand, it is also naturally extensible:

use textwrap::Indented;
use owo_colors::OwoColorize;
use num_format::{Locale, ToFormattedString};

println!("{text.indented(4).red()}");
println!("{number.to_formatted_string(&Locale::de)}");

The benefits of this approach are clear:

It is regular Rust syntax that works with IDE autocompletion, documentation tooltips, syntax highlighting, etc.
Rather than a symbol, it has a descriptive name that’s easy to understand
It is a zero-cost abstraction: What you don’t use, you don’t pay for.
Formatting functions don’t have to be implemented in the standard library, the can live in 3rd party crates.
Formatters can be easily combined.

Only {:?} and {:#?} we should keep, because these options are used very often, and concise syntactic sugar is important for debugging.

A new approach

It is tempting to just desugar f"…" to format_args!("…") (which doesn’t allocate a String), but then template strings would inherit the complex formatting machinery, which we don’t want. So let’s go back to the drawing board. Here’s what we need:

A way to display text to the user, formatted in various ways
A way to print values and data structures for debugging

Today, these use cases are mostly handled by the Display and Debug traits. I expect that Debug will continue to be used, and {…:?} and {…:#?} will be available in template strings. However, compile-time reflection (which is a project goal for 2025) will hopefully make #[derive(Debug)] obsolete, so debug printing Just Works™.

The `Pretty` trait

For user-facing formatting, I propose a new trait to supersede Display:

trait Pretty {
    fn fmt(&self, f: &mut core::pretty::Formatter) -> core::pretty::Result;
}

The difference to Display is that this Formatter doesn’t include any formatting options. It just provides the following methods:

write(impl Pretty)
writeln(impl Pretty)
write_char(char)
write_str(&str)
write_display(impl Display) for interoperability

Here is an example how Pretty could be implemented for a type representing a parser warning:

impl pretty::Pretty for ParseWarning {
    fn fmt(&self, f: &mut pretty::Formatter<'_>) -> pretty::Result {
        f.writeln(self.warning)?;
        f.writeln(f"  at {self.file()}@{self.range()}")?;
        if let Some(note) = &self.note {
            f.writeln(f"  {"note".blue()}: {note.pretty()}")?;
        }
        Ok(())
    }
}

Because write() and writeln() accept any type implementing Pretty, we can also give it a template string, which is more convenient than having to use the write! macro.

With a blanket Display implementation for all Pretty types, migration should be smooth. To ease migration, the old std::fmt::Formatter can also get a write_pretty method.

Supporting `no_std`

Making the Pretty trait interoperable with Display poses a problem: To make it work, the core::pretty::Formatter also needs to use &dyn std::fmt::Write under the hood, which is impossible while the latter is not in core. But it should be possible to move std::fmt::Write to core, since it doesn’t require allocation and the Error type is just a unit struct. But that’s the least fleshed-out part of this idea.

If you enjoy reading about language design, let me know. Discussion on Reddit.

Effects in Rust (and Koka)

Fri, 10 Oct 2025 00:00:00 GMT

What is an effect system? According to Wikipedia, it is a formal system that describes the computational effects of computer programs, such as side effects. It is also an extension of a type system, and allows you to statically verify that your program is sound with regard to effects.

If you want to fully understand this concept, I recommend you to learn Koka. It is a beautiful language with a powerful, yet easy to understand effect system. This blog post includes Koka snippets, but they should be easy to understand without prior knowledge.

Side effects in Rust

The most common kind of effect is a side effect. That’s when a function changes state that is observable outside of the function, other than through the return value. For example, printing text to stdout is a side effect, and so is mutating a global variable.

Despite being imperative at its core, Rust has borrowed many ideas from functional programming. This is not just because FP is ‘cool’, but because Rust aims to make your code more reliable, maintainable, testable and easy to understand. In particular, programmers praise the fact that Rust can prove the absence of data races in multi-threaded code, which is notoriously difficult in other imperative languages. Rust achieves this through its ownership model, which limits how and when data can be mutated, and this is encoded in the type system:

fn report_items(items: Vec<&mut Item>) {
    for item in items {
        println!("{item}");
        item.reported = true;
    }
}

One could argue that the ownership model is an effect system (albeit an incomplete one), in which &mut declares a potential side effect by mutation. Side effects not tracked by Rust’s type system are

Memory allocation
Input/Output (e.g. networking, file system access)
Mutation through interior mutable types or raw pointers

This is by design: Rust may be dogmatic about safety, but it is much more pragmatic when it comes to programming idioms; it is not a purely functional language.

Effects that modify control flow

There’s another kind of effect that can suspend and potentially resume a function, which includes

panic
await
yield
? ¹

Panics (exceptions)

Rust doesn’t have exceptions, but panics are quite similar: They bubble up, unwinding the call stack and calling destructors in the process, until they’re caught (unless you set panic=abort). Whereas exceptions are typically caught with a try/catch construct, Rust’s panics are caught with the catch_unwind function.

The main difference between exceptions and panics is that panics are not the default choice for error handling. They are intended only for errors that the program cannot recover from, and should only be caught in order to log or report them. Instead, the Result type is used for typical error handling.

This means that errors are not side effects in Rust. Result::Err(_) is returned as a value, which makes it indistinguishable from a successful return in the eyes of the type system. Similarly, ? is not a side effect, as it merely desugars to an early return statement. But errors are algebraic effects, which are part of the type system. This ensures that all possible errors are handled, otherwise your code doesn’t compile.

Now, let’s take a look at how exceptions are implemented in Koka. Koka is a functional programming language with a simple yet powerful effect system. Rather than having many forms of control flow built in, it provides a few basic building blocks for implementing all kinds of control flow:

effect throw
  ctl throw(msg : string) : a

fun safe-divide(x : int, y : int) : throw int
  if y == 0 then throw("div-by-zero")
  else x / y

What’s going on here? We declare an effect throw that accepts a string and returns an arbitrary type a (in Rust, we would use the ! type here). This effect is called in the safe-divide function, and becomes part of its type signature: throw int means that this function returns int and has the effect throw.

The ctl keyword means that this is a control flow effect, meaning that it interrupts the execution (like an exception). Control flow effects can also resume the execution later.

When safe-divide is called, the effect must be either handled, or propagated:

fun propagate() : throw int
  safe-divide(4, 0)

fun handle-effect() : int
  with handler
    ctl throw(msg) 42
  propagate()

Now here comes the interesting part: handle-effect calls propagate, which calls safe-divide, which invokes the throw effect. The handler expression, well, handles the effect by returning 42. In Rust, this would be written as

fn handle_effect() -> i32 {
    std::panic::catch_unwind(propagate)
        .unwrap_or_else(|_| 42)
}

When an effect is invoked in Koka, the current execution is suspended. The effect handler may resume the execution, but it doesn’t have to.

Await and yield

Generators are an unstable feature in Rust that makes it easy to write iterators:

let generator = gen {
    yield 5;
    yield 3;
    yield 1;
};

for n in generator {
    println!("yielded {n}");
}

The interesting part is the yield keyword. It emits a value and pauses the generator, which is then resumed once the iterator’s next() method is called. In Koka, this could be written like this:

effect yield
  ctl yield(item : t) : ()

fun generator() : yield ()
  yield(5)
  yield(3)
  yield(1)

fun main()
  with handler
    ctl yield(item)
      println("yielded " ++ item.show)
      resume()
  generator()

This looks very similar to the exception handling example, with one important difference: After handling the yield effect, we call resume(), which continues the execution of the generator.

Async/await is similar, but more powerful. An await expression pauses the execution, but it also produces a new value:

effect await
  ctl await(fut : future<t>) : t

This value can be provided to the resume() function when handling the await effect. I won’t attempt to implement this, but it is most certainly possible.

Effect Polymorphism

What makes effect systems so powerful is their ability to be polymorphic over effects. Polymorphic means having several shapes or forms. For example, Koka allows you to define a function

fun map(xs : list<a>, f : a -> e b) : e list<b>

Where e is a generic effect, or any number of effects. This higher-order function accepts functions with arbitrary effects and simply propagates them.

The same is not possible in Rust: To support await, the function must be async, and to support yield, it must return a generator. However, if you decide to support await and make the function async, it cannot be used synchronously anymore, whether or not await is actually used. For this reason, some libraries reluctantly offer two APIs: one synchronous and one asynchronous.

There’s another case where Rust’s lack of effect polymorphism causes problems: sometimes we want to produce an error from a higher-order function, but the function doesn’t support this. This has led to a proliferation of functions such as try_map, try_find, try_fold and try_reduce. As you might recall, errors in Rust aren’t side effects because they are returned as values. This is generally a good approach because it makes errors subject to the type system. However, it can be difficult to work with at times due to the inability to be polymorphic over fallible and non-fallible functions.

Keyword Generics

In 2022, the Rust project announced the Keyword Generics Initiative, which aims to add support for effect polymorphism. In particular, the initiative wants functions (and traits) to be generic over their async-ness, const-ness, fallibility, and perhaps the mutability of self. I previously believed that the initiative had been abandoned, given that no comprehensive proposal was created. However, there is still sporadic activity in tracking issues.

Constness

What’s the deal with const? Since it doesn’t produce side effects or change control flow, const cannot be considered an effect. It is actually the opposite: The absence of const – let’s call it runtime – is an effect. Pure functions can in theory always be made const, whereas functions with side effects other than panics always require the runtime effect; they cannot be const.

There is a nice symmetry between runtime and async: Both effects are “infectious”, meaning an async function cannot be called from an synchronous function, just as a runtime function cannot be called from a const function. The other direction works: A synchronous function can be called from an async function with no problem. The same goes for const functions.

Right now, the Rust project is working hard to make traits usable in const functions. This requires some form of effect polymorphism:

// currently proposed syntax

[const] trait Default {
    fn default() -> Self;
}

struct Thing<T>(T);

impl<T: [const] Default> const Default for Thing<T> {
    fn default() -> Self { Self(T::default()) }
}

impl const Default for () {
    fn default() {}
}

The [const] modifier means “this might be const, or it might not”. Let’s see what this would look like in Koka. Since Koka doesn’t have traits or type classes, we use implicit parameters (denoted by ?) instead:

struct thing<a>
  value : a

fun thing/default(?default : () -> e a) : e thing<a>
  Thing(default())

fun unit/default()
  ()

This is more powerful, since it works with any effect – not just runtime. Other than that, it does the same thing as the Rust example. The ? means that the default parameter can be omitted and Koka will find the correct function automatically.

Encoding side effects in the type system

Many effects can already be expressed in Rust. An effect that does not suspend execution (i.e. one that immediately resumes) is equivalent to a function argument.

fun count-lines(path : path) : <exn,fsys> int
  read-text-file(path).sep/split("\n").length

This can be written in Rust as

fn count_lines(
    path: &Path,
    read_text_file: impl Fn(&Path) -> io::Result<String>,
) -> io::Result<usize> {
    Ok(read_text_file(path)?.split("\n").count())
}

The side effect (file I/O) is now encoded in the function signature. This has several advantages: it makes reasoning about the program easier and simplifies unit testing. To make it scale better, a trait can be used:

trait Fsys {
    fn read_text_file(&self, path: &Path) -> io::Result<String>;
}

fn count_lines(path: &Path, fsys: impl Fsys) -> io::Result<usize> {
    Ok(fsys.read_text_file(path)?.split("\n").count())
}

The main disadvantage is that this parameter has to be explicitly passed down from main to every function that accesses the file system. If a function has multiple effects, this can become tedious. However, having too many functions with side effects is an anti-pattern, and making them explicit can help you to refactor and improve your code.

Conclusion

Effect.ts is a new and popular framework enabling functional programming, including an effect system, in TypeScript. This demonstrates the growing demand for type and effect safety within the software engineering industry.

Unfortunately, a fully general effect system like Koka’s most likely cannot be added to Rust without major breaking changes. To me, the proposed keyword generics feel like a compromise rather than a shiny future, but they are still a big improvement over the status quo. In the meantime, you can use parameters to make side effects explicit. And if you haven’t already, take a look at Koka.

Discuss this blog post on Reddit.

I previously claimed that Rust’s errors aren’t effects, which is only partially correct: Errors aren’t side effects, as they are part of the return type, but they are algebraic effects. ↩

We don't need revenge

Tue, 30 May 2023 00:00:00 GMT

Mob justice is when a person suspected to be a criminal is beaten by a group of people or crowd with clubs, stones, machetes, or anything they can lay their hands on — Wikipedia

Is this what the Rust community wants?

If you don’t know what I mean, I’m talking about this Reddit thread, which asks for courage and accountability from the Rust project leadership. That sounds reasonable, who doesn’t want the leaders to be accountable? The problem is in how they want to achieve it: By asking Rust project members to “name and shame” the people who messed up. But what happens then?

Background

It started with this blog post, where ThePHD explained why they are no longer speaking at RustConf 2023. Some people from the Rust project had reservations about ThePHD’s work, but it’s unclear who exactly. Since then, a shitstorm has unfolded as more and more alarming details have come to light. Someone in the Rust leadership even resigned in protest, and a well-known Rust educator also ended his involvement. Now, the Rust project leadership has apologized and said that the people responsible have resigned from their positions. Of course, people want to know who in the Rust project was responsible.

Accountability, whatever that means

The Rust community has always been perceived as happy and welcoming, and we are proud of that. We have made it possible for anyone, regardless of background, to learn the language, contribute, and propose new features. Changes must follow the RFC process, which is based on consensus, and communication happens in the open… mostly. Some decisions are time critical or involve personal/confidential information, so we don’t know everything the Rust leadership does.

What does accountability mean? According to Wikipedia, it means “A is accountable to B when A is obliged to inform B about A’s (past or future) actions and decisions, to justify them, and to suffer punishment in the case of eventual misconduct” and more.

Accountability is important to ensure that people act responsibly. But to whom should Rust’s leadership be accountable? There are three options:

They are accountable to each other
They are accountable to another person or group, like a control gremium
They are accountable to the wider Rust community

The last option doesn’t really make sense though: When an event or decision affects the public, the public should be informed in the interest of transparency. But when someone does something wrong, the public is not in a position to punish the person appropriately, or to fix the problem.

The ethics of punishment

Every government has a justice system for punishing criminals, even though we don’t know exactly what “justice” means. People seem to have an innate sense of justice, but it is coupled with our emotions. To justify punishment, it must be useful, and of course there are quite a few reasons why punishment is useful:

It prevents the criminal from doing it again
It discourages others from doing anything illegal
The criminal gets to repent and feel less guilty after accepting the punishment
The victim may feel vindicated by getting “revenge”

But before punishing someone, we must first understand why the criminal did it. Most people (including criminals) are not evil psychopaths, and understanding their perspective is important to figure out what to do with them. Therefore, a criminal has the right to a fair trial in any Rechtsstaat (legal system).

Now, the Rust project is not a government, but it can still punish someone by banning them or removing them from their leadership position. Of course that person should be given a chance to justify their actions. When someone behaves badly, it’s usually an accident, often caused by bad communication.

Why do we need to know whose fault it was?

We don’t, because it wouldn’t help anyone. I would support it if the people who treated ThePHD harshly apologized on their own terms. But outing them without their permission is not acceptable: It puts them at risk of online shaming, which goes against everything that Rust stands for. And even though online shaming can be seen as a punishment for their actions, there are two problems:

Without a trial, we don’t know what punishment they deserve.
A punishment should never violate someone’s dignity, no matter how serious the crime.

As long as we don’t have all the information, all we can do is speculate (which doesn’t mean that we should). Unsurprisingly, the people within the project, who do have all the information, already took steps to resolve the conflict and punish those who messed up. The Rust project isn’t perfect, no organization consisting of humans is. But I do think they are trying their best.

A zero-overhead linked list in Rust

Mon, 12 Apr 2021 00:00:00 GMT

Let’s implement an immutable, singly-linked list. Singly-linked means that each node contains a reference to the next node, but not vice versa. To make this data structure really performant, let’s use plain references instead of heap-allocated types. This would be dangerous in memory-unsafe languages like C, because it could easily cause vulnerabilities because of dangling pointers, but Rust’s lifetimes protect us from this. We’ll see what this means in a moment.

Implementation

The implementation is fairly simple:

pub enum List<'a, T> {
    Node {
        data: T,
        next: &'a List<'a, T>,
    },
    Tail
}

So a list is either a node containing some data and a reference to the next node, or the tail, i.e. the end of the list. Let’s write a Default implementation that creates an empty list, which is just List::Tail:

impl<T> Default for List<'_, T> {
    fn default() -> Self {
        List::Tail
    }
}

Now how do we add elements to the list? As I mentioned above, this list is immutable, so altering a list is not possible. Instead, let’s introduce a concept from functional programming: Persistent data structures

Persistent data structures

According to wikipedia:

A persistent data structure is a data structure that always preserves the previous version of itself when it is modified. Such data structures are effectively immutable, as their operations do not (visibly) update the structure in-place, but instead always yield a new updated structure.

We can achieve this by prepending a node at the front of the list: The previous list stays the same, but we get a new node pointing to it.

impl<'a, T> List<'a, T> {
    pub fn add(&'a self, data: T) -> Self {
        List::Node { data, next: self }
    }
}

Let’s try it out:

#[test]
fn test_add() {
    let tail = List::Tail;
    let first = tail.add(5);
    let second = first.add(10);

    assert_eq!(tail, List::Tail);
    assert_eq!(first, List::Node {
        data: 5,
        next: &List::Tail,
    });
    assert_eq!(second, List::Node {
        data: 10,
        next: &List::Node {
            data: 5,
            next: &List::Tail,
        },
    });
}

For this to work, we need to implement Debug and PartialEq for our type. While we’re at it, let’s also implement some other useful traits. Note that we can even implement Copy, since next is just a reference, not a Box or Rc:

#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum List<'a, T> {
    // snip
}

That’s it! Now we can test it:

> cargo test -q

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00

Writing an iterator

In a functional language, we’d use this data structure mainly in recursive functions. It is, after all, a recursive data structure. But since this is Rust, let’s implement an Iterator for our type:

pub struct ListIter<'a, T>(&'a List<'a, T>);

impl<'a, T> Iterator for ListIter<'a, T> {
    type Item = &'a T;

    fn next(&mut self) -> Option<Self::Item> {
        match self.0 {
            List::Node { data, next } => {
                self.0 = next;
                Some(data)
            }
            List::Tail => None,
        }
    }
}

We can also implement IntoIterator to make iterating over a List easier:

impl<'a, T> IntoIterator for &'a List<'a, T> {
    type Item = &'a T;

    type IntoIter = ListIter<'a, T>;

    fn into_iter(self) -> Self::IntoIter {
        ListIter(self)
    }
}

Let’s test this as well:

#[test]
fn test_iterator() {
    let tail = List::Tail;
    let first = tail.add(5);
    let second = first.add(10);

    let values: Vec<i32> =
        second.into_iter().copied().collect();

    assert_eq!(values, vec![10, 5]);
}

But wait! Why are the values in the opposite order of how we added them? That’s because the add function prepends nodes at the start of the list. We can think of it like a stack of books: We can put books on top of it, and we can take books from it, but only in the opposite order.

Can’t we just use .rev() to iterate in reverse direction? Let’s try it:

let values: Vec<i32> =
    second.into_iter().rev().copied().collect();

error: the trait bound `ListIter<'_, {integer}>: DoubleEndedIterator` is not satisfied
  --> src/lib.rs:78:28
   |
 2 |     second.into_iter().rev().copied().collect();
   |                        ^^^ the trait `DoubleEndedIterator` is not implemented for `ListIter<'_, {integer}>`

So it doesn’t work. To make it work, we’d have to implement the DoubleEndedIterator trait, but that can’t be done very efficiently, because a list node can’t access its previous node. Then let’s try a different approach!

Internal iteration

Iterating over the list in reverse order can be done efficiently, if we use a recursive algorithm. This doesn’t work with the design of Iterator trait, so we’ll just implement it as an internal iterator, i.e. a function that accepts a closure:

impl<'a, T> List<'a, T> {
    // snip

    pub fn rev_iter(&'a self, f: impl Fn(&'a T)) {
        if let List::Node { data, next } = self {
            next.rev_iter(&f); // 1
            f(data); // 2
        }
    }
}

We call rev_iter recursively.
Because we want to iterate in reverse, we call the closure after the recursive function call.

This has the downside that we can’t use iterator combinators like filter or map. It also doesn’t allow error handling in the closure, but we can add one more internal iterator that stops iterating when the closure returns an error:

impl<'a, T> List<'a, T> {
    // snip

    pub fn try_rev_iter<E, F>(&'a self, f: F) -> Result<(), E>
    where
        F: Fn(&'a T) -> Result<(), E>,
    {
        if let List::Node { data, next } = self {
            next.try_rev_iter(&f)?;
            f(data)?;
        }
        Ok(())
    }
}

I’m omitting the test here, but you can read it in the playground, or you can implement it as an exercise.

Limitations

This type of list isn’t useful very often. One reason for this is that it can only be iterated over by-reference. The other reason I’ll demonstrate now. Let’s try to create a function that constructs a List and returns it:

fn too_bad<'a>() -> List<'a, i32> {
    let tail = List::Tail;
    let v1 = tail.add(1);
    let v2 = v1.add(2);
    v2
}

error[E0515]: cannot return value referencing local variable `tail`
   --> src/lib.rs:107:5
    |
  3 |     let v1 = tail.add(1);
    |              ---- `tail` is borrowed here
  4 |     let v2 = v1.add(2);
  5 |     v2
    |     ^^ returns a value referencing data owned by the current function

error[E0515]: cannot return value referencing local variable `v1`
   --> src/lib.rs:107:5
    |
  4 |     let v2 = v1.add(2);
    |              -- `v1` is borrowed here
  5 |     v2
    |     ^^ returns a value referencing data owned by the current function

So this doesn’t work. We can’t return a list if it was created in the current function. How about a function that adds elements to a list in-place?

fn doesnt_work<'a>(list: &'a mut List<'a, i32>) {
    let old_list = std::mem::take(list);
    *list = old_list.add(5);
}

error[E0597]: `old_list` does not live long enough
   --> src/lib.rs:112:13
    |
  1 | fn doesnt_work<'a>(list: &'a mut List<'a, i32>) {
    |                -- lifetime `'a` defined here
  2 |     let old_list = std::mem::take(list);
  3 |     *list = old_list.add(5);
    |             ^^^^^^^^-------
    |             |
    |             borrowed value does not live long enough
    |             argument requires that `old_list` is borrowed for `'a`
  4 | }
    | - `old_list` dropped here while still borrowed

That doesn’t work either. This probably doesn’t come as a surprise if you’re familiar with Rust’s ownership rules: When we return a list, it can’t borrow anything defined in the current function. If Rust didn’t prevent this, we could accidentally get a dangling reference.

Use case

So when is this type useful? When you need to efficiently add nodes to a list without modifying the original list, and the limitations above are not a problem.

One example that comes to mind is an interpreter for a stack-based programming language. Here’s a simple example:

#[derive(Clone, PartialEq)]
pub enum Value {
    Num(f64),
    Bool(bool),
    String(String),
}

pub enum Expr {
    Value(Value),
    Variable(String),
    UnExpr(UnExprKind, Box<Expr>),
    BinExpr(BinExprKind, Box<(Expr, Expr)>),
    Define(String, Box<(Expr, Expr)>),
    IfThenElse(Box<(Expr, Expr, Expr)>),
}

pub enum UnExprKind {
    Not,
    Neg,
}

#[derive(PartialEq)]
pub enum BinExprKind {
    // Arithmetic
    Add,
    Sub,
    Mul,
    Div,

    // Logic
    And,
    Or,
    Equals,
    NotEquals,
}

This programming language has three data types, numbers, booleans and strings. Numbers support basic arithmetic operations (+, -, *, /), booleans support logic operations (&&, ||, !, ==, !=). Operations are divided into unary operations (those with only one operand) and binary operations (those with two operands).

The programming language is expression-based, so everything is an expresssion. An expression can be a value, a variable, a unary or binary operation, a variable definition, or a condition. Note that a variable definition always introduces a new scope in which the defined variable can be used. The syntax could look something like this:

define x = 5 in
    some_expression

This defines a variable x, which can be used in some_expression. The only way to define multiple variables is to nest them:

define x = 5 in
    define y = 42 in
        x + y

Now let’s implement the function that evaluates this language:

type Variables<'a> = List<'a, (String, Value)>;

pub fn eval(
    vars: &Variables<'_>,
    expr: Expr,
) -> Option<Value> {
    match expr {
        Expr::Value(val) => Some(val),

        Expr::Variable(var) => vars  // 1
            .into_iter()
            .find(|&(v, _)| *v == var)
            .map(|(_, val)| val.clone()),

        Expr::UnExpr(kind, expr) => {
            eval_unary(kind, vars, *expr)
        }

        Expr::BinExpr(kind, exprs) => {
            eval_binary(kind, vars, exprs.0, exprs.1)
        }

        Expr::Define(name, exprs) => {
            let value = eval(vars, exprs.0)?;
            let vars = vars.add((name, value));  // 2
            eval(&vars, exprs.1)
        }

        Expr::IfThenElse(exprs) => {
            if let Value::Bool(b) = eval(vars, exprs.0)? {
                eval(vars, if b { exprs.1 } else { exprs.2 })
            } else {
                None
            }
        }
    }
}

fn eval_unary(
    kind: UnExprKind,
    vars: &Variables<'_>,
    expr: Expr,
) -> Option<Value> {
    match (kind, eval(vars, expr)?) {
        (UnExprKind::Not, Value::Bool(b)) => {
            Some(Value::Bool(!b))
        }
        (UnExprKind::Neg, Value::Num(n)) => {
            Some(Value::Num(-n))
        }
        _ => None,
    }
}

fn eval_binary(
    kind: BinExprKind,
    vars: &Variables<'_>,
    lhs: Expr,
    rhs: Expr,
) -> Option<Value> {
    match kind {
        BinExprKind::Add => {
            if let Value::Num(lhs) = eval(vars, lhs)? {
                if let Value::Num(rhs) = eval(vars, rhs)? {
                    return Some(Value::Num(lhs + rhs));
                }
            }
            None
        }
        // remaining match arms omitted
    }
}

Look up the variable with the name var.
Add a new variable to the list

The full code, including tests, is in the playground.

Why is a linked list better than a Vec in this case? Because a variable should only be visible in its own scope, so when evaluating a variable definition, the list of variables should be the same afterwards as before.

One way to achieve this with a Vec is to clone it whenever a new variable is added, but this is quite inefficient.

An alternative is to add the variable to the Vec and remove it again when the variable’s scope ends. The problem with this is that it requires passing a mutable Vec around, so the type system can’t ensure that the Vec’s previous state is restored after a variable definition. A small error or even an early return could break it. This could be prevented with a RAII guard, but the solution using List is more elegant.

Alternatives

To get around the lifetime issues, you can use reference-counted smart pointers (Rc or Arc):

enum List<T> {
    Node {
        data: T,
        next: Arc<List<T>>,
    },
    Tail,
}

This has all the advantages of our type, except that Rc and Arc incur a slight performance overhead when creating, cloning or dropping it.

When you don’t need the list to be persistent or immutable, you can just use a Vec instead. This also has some overhead due to heap allocations, but in return has better cache locality. More importantly, it’s easier to use: Keeping your code maintainable and easy to read is usually more important than to squeeze out every last drop of performance. Heavy optimization is only needed in performance-critical sections of the code.

The often-cited quote “Premature optimization is the root of all evil” doesn’t mean that you shouldn’t optimize, rather that you should be clever about it. Do the most effective things to improve performance first: Choose efficient algorithms. Then benchmark your code, identify where it spends most of its time, try to optimize these parts, and verify that your optimizations actually yield an improvement. Trying to optimize code without measuring the results is a hopeless endeavor.

Fin

I’m looking forward to the discussion on Reddit. You can also open an issue in the issue tracker. Until next time!

Rusts Module System Explained

Sun, 28 Mar 2021 00:00:00 GMT

The Rust programming language can be confusing for beginners, and the module system is one part that causes frustration particularly often. There are quite a few blog posts out there trying to explain the module system in a simple way, but I often have the feeling that they over-simplify things. So here’s my take—a more detailed explanation of the module system.

This post assumes that you can write at least a “hello world” Rust program. It’s a rather long read, so get comfortable, maybe with a cup of tea, hot chocolate, or whatever your heart desires 😊

Why do modules exist?

Modules give your code structure: Dividing your code into modules is like dividing your house into several rooms: Each room has a different purpose, and rooms can be locked for privacy.

Floor plan. Source (modified; license)

The module tree

Modules are structured in a hierarchy, the module tree, which is similar to the file tree of the source files. There are two kinds of modules: Inline modules and “normal” modules:

mod inline {
    // content of the module
}

mod normal;
// the content is in another file

These are functionally equivalent. If the content of an inline module is very long, move it to another file, to keep the code neat and manageable.

When the module is not inline, Rust looks for the content of the module in another file, either module_name.rs or module_name/mod.rs. It might seem odd that we have to declare modules explicitly (unlike in Python, where modules are inferred from the file system). However, there are good reasons for this, as we’ll see later.

Like every tree, the module tree has a root. This is the file lib.rs in case of a library crate, or the file main.rs in case of a binary crate ¹.

Submodules

Unfortunately, Rust is not the most consistent language when it comes to modules: There are two different ways to structure a module tree, and they can be mixed within the same crate.

Say we have a library crate with a module parent, which contains a sub-module child:

└─ library root
   └─ parent
      └─ child

The crate root is in a lib.rs file in the src directory. However, the parent module can be either in a parent.rs file next to lib.rs, or in a mod.rs file in a parent directory:

File tree A
File tree B

Cargo.toml
src/
- lib.rs
- parent.rs the parent module
- parent/
  - child.rs

It doesn’t really matter which way you go, just do what you prefer. I use the first way (“File tree A”), since it’s easier to add sub-modules. For example, if you want to add a submodule to child, you just need to create a folder and a new file, and add a mod declaration:

Cargo.toml
src/
- lib.rs
- parent.rs
- parent/
  - child.rs
  - child/
    grand_child.rs

The path of a module can also be specified explicitly with the #[path] attribute, but this is rarely used in practice.

An example

Hopefully this will make more sense once you see an example. Here’s the module structure of a library crate:

Cargo.toml
src/
- lib.rs
- foo.rs
- bar.rs
- bar/
  - baz.rs

lib.rs

// root module

pub mod foo;
pub mod bar;

bar.rs

pub mod baz;

pub use baz::*;
pub use crate::foo::Answer;

foo.rs

mod answer {
    pub struct Answer(pub i32);
}

pub use answer::Answer;

baz.rs

use super::Answer;

pub fn answer() -> Answer {
    Answer(42)
}

No worries if you don’t understand everything here! All the concepts that are used here will be explained. You can look at this example later and see if you understand everything. For now, do you know what the module tree is?

See solution

└─ library root  /src/lib.rs
   ├─ foo        /src/foo.rs
   │  └─ answer  /src/foo.rs
   └─ bar        /src/bar.rs
      └─ baz     /src/bar/baz.rs

Items and paths

A module contains items. Items are

Functions
Types (structs, enums, unions, type aliases)
Traits
Impl blocks
Macros
Constants and statics
Extern blocks
Extern crates
Imports
Modules
Associated items (not important right now)

You can refer to items by their path. For example, the path foo::bar::Baz refers to the Baz item within the bar item within the foo item. Paths are usually relative: To use foo::bar::Baz, the foo item must be available in the current scope; absolute paths (starting at the root module) are prefixed with crate::. A super:: path segment changes to the parent module (similar to ../ in the file system).

Imports are used to shorten paths. Instead of having to write foo::bar::Baz every time, we can write use foo::bar::Baz; once. This brings the item into scope, so we can refer to it with the much shorter path Baz.

Visibility

Visibility, or privacy, is the concept of making parts of a module inaccessible from other modules. Things that are only accessible in the same module are called private, and things that are accessible everywhere are called public.

In Rust, most things are private by default. To make something public, the pub keyword is written before it. This makes the item accessible everywhere:

lib.rs

mod foo { // 1
    pub mod bar {
        struct Baz;
    }
    // use bar::Baz; // 2
}

use foo::bar; // 3

This declares a private module, so it can only be used within this root module. It can’t be accessed from another crate.
If we uncommented this, it would fail to compile. Baz is private, therefore it can only be used within the bar module.
The module bar can be used here, because it is declared as public. This is somewhat counter-intuitive, since the foo module is private. But when a module is private, it can still be accessed within its direct parent module, since a module is just like any other item.

Encapsulation

When designing an API, there are often invariants that need to be preserved. An invariant is a property that never changes. For example, a struct might contain a value that is supposed to always be within the interval [0; 360):

pub struct Angle(pub f32);

Let’s write a new function that validates this invariant, and a getter for the value:

impl Angle {
    pub fn new(value: f32) -> Self {
        Angle(value.rem_euclid(360.0))
    }

    pub fn value(&self) -> f32 {
        self.0
    }
}

rem_euclid calculates the least nonnegative remainder of self (mod rhs).

By ensuring that the angle is always in [0; 360), we can implement operations such as equality (where 0° == 360°) very easily. But wait! Since the field is public, a user of the API can create an Angle object without calling the new function, or modify it without checking the invariant.

By making the field private, the struct’s implementation details are hidden. This is called encapsulation: Within this module, we still have to take special care that the invariant is preserved, but if the code is correct, the public API is impossible to use incorrectly.

Fine-grained visibility

Items can be private or public. However, there are also visibilities in-between: Most notably, an item can be declared as pub(crate). This means that it is visible within the current crate, but not outside. With pub(super), an item is visible within the parent module. With pub(in path), visibility can also be limited to any other module as well:

pub(crate) mod foo {
    pub(super) fn bar() {}
    pub(in crate::foo) struct Baz;
}

When something is visible in one module, it is also visible in all its child modules. It still needs to be imported (or referred to with its path) though:

struct Foo;
// Foo is visible in this module

mod inner {
    use super::Foo;
    // Foo is also visible here!
}

Visibilities overview

Visibility	Description
`pub`	The item is visible everywhere
`pub(crate)`	The item is visible in the current crate
`pub(super)`	The item is visible in the parent module
`pub(in some::path)`	The item is visible in the specified path. The path must refer to an ancestor module of the item.
`pub(self)`	The item is private, i.e. visible only in the same module. This is equivalent to omitting the visibility entirely.

Exports

With pub use declarations, items can be re-exported from a different module than the one they were declared in. A re-exported item has multiple paths that refer to the same thing. For example:

lib.rs

pub mod answer {
    pub const ANSWER: i32 = 42;
}
pub use answer::ANSWER;

Now ANSWER can be referred to as either crate::ANSWER or crate::answer::ANSWER. However, not every path is always reachable. Take, for example:

lib.rs

mod answer {
    pub const ANSWER: i32 = 42;
}
pub use answer::ANSWER;

crate::answer::ANSWER is public, but it can’t be used from outside the crate, because the answer module is private. Only the re-export crate::ANSWER can be used from outside the crate.

Common pitfalls

The module tree must be built manually.

There’s no implicit mapping from the file system tree to the module tree: We need to declare all modules with the mod keyword.

Don’t confuse visibility with reachability.

The visibility of an item is like an upper bound, it can’t be increased with re-exports. For example, we can’t re-export a private struct outside of its module.

However, a public item might not be reachable from outside the crate, if it’s in a private module and isn’t publicly re-exported. To make an item available in the crate root, it’s not enough to make it public; we also need to make it reachable from the crate root.

Don’t confuse visibility with availability.

Visibility means that we are principally allowed to use an item somewhere. It doesn’t mean that the item is available, i.e. in scope, so we might still have to import it (or refer to it with its path).

Special cases

There are a few language constructs that don’t adhere to the same rules as everything else:

Enum variants and fields

Enum variants and variant fields are always public, and it’s not possible to make them private. Therefore we should be careful when exposing enums publicly, because changing the variants or fields later is not backwards compatible.

You can add the #[non_exhaustive] attribute to an enum to allow adding more variants later. This mean that the enum can’t be matched exhaustively; we’ll always need to add a wildcard match arm (_ => {}).

Sometimes it’s a good idea to wrap an enum in a struct to hide the implementation details:

pub struct Foo(FooImpl); // FooImpl is private

enum FooImpl {
    // ...
}

Also, when an enum variant has multiple fields, it’s usually better to put them in a separate struct, so it’s possible to make the fields private or make the struct non_exhaustive:

// Instead of this:
pub enum Foo {
    Variant {
        field: i32,
        other_field: bool,
    }
}

// do this:
pub enum Foo {
    Variant(FooVariant),
}
pub struct FooVariant {
    field: i32,
    other_field: bool,
}

Macros

Declarative macros (the ones that are declared with macro_rules!) behave more like local variables within a function than like items in some regards. For example, they can be shadowed, they have to be declared before they can be used, and they don’t need to be explicitly imported in child modules ².

And, they can’t be declared public. The #[macro_export] attribute can be added to a macro, which exports it publicly at the crate root. This can be undesirable, however, if it’s not supposed to be part of the public API; there is no equivalent of pub(crate) for macros.

One workaround for this is to put our macros in a module and annotate the module with #[macro_use]. The module should be the first module declaration in the crate root. This ensures that the macros can be used everywhere in our crate, but not outside of the crate. Not the most elegant solution, but it works.

Why are modules declared explicitly?

I promised to explain why modules have to be declared explicitly. There are a few reasons:

With the #[path] attribute, a module can be located in a different directory, or have a different name than the file.
Module declarations can have a visibility, e.g. pub(crate) mod foo;
Sometimes there are files which you don’t want to include in the module tree.

For example, a crate with both a library and a binary target usually contains a lib.rs file for the library and a main.rs file for the binary. Submodules are stored in the same directory, but some modules are only needed by the library, and some only by the binary. By specifying the modules explicitly, you can include only the necessary modules in each file.

Fin

I hope you liked this post! Please let me know if you found this article useful; were there any things that were unclear or confusing? I’ll try to improve it over time.

Discussion on Reddit. You can also open an issue in the issue tracker.

Until next time!

A crate can also have multiple targets (library, binary, example, test, and benchmark targets), in which case each target has its own root. You can read more about this here. ↩
This is called “textual scoping”. Actually, macros can have both a textual scope (like local variables) and a path-based scope (like items); the rules for this are quite complicated. ↩

Implementing RAII guards in Rust

Thu, 18 Mar 2021 00:00:00 GMT

As you probably know, Rust doesn’t have automatic garbage collection. Instead, it relies on destructors to clean up memory, and these destructor calls are automatically inserted in the appropriate places at compile time. And since Rust uses traits for everything, destructors use a trait as well: The Drop trait.

This has the benefit that Drop can be implemented to do anything, not just cleaning up memory: It can also release a file descriptor, close a network socket, cancel an in-flight HTTP request, and much more. Whenever you acquire some sort of resource, and want to release it when you’re done with it, the Drop trait is your friend.

In this blog post we’ll first look at how the Drop trait works, and then implement the RAII guard pattern, step by step.

Where is `Drop` being used?

The most common example are heap-allocated containers, such as Box, Vec and String. When such a value goes out of scope, it gets dropped, which means that its Drop implementation is called:

fn main() {
    // String is allocated on the heap
    let mut s = "hello, world!!".to_string();
    do_something(&mut s);
} // `s` is dropped, heap allocation is freed

Another use case are containers that provide interior mutability, such as RefCell or Mutex. Before we can mutate one of these types, we have to aquire a write lock. Once the write lock is released, it can be aquired again. It is guaranteed that there is at most one write lock at any time, to prevent data races footnote:[RefCell can’t actually prevent data races, it only prevents aliasing mutable references, which would violate the ownership rules; RefCell is not thread-safe (it doesn’t perform locking), so Rust ensures that it can only be used in a single thread. You can use a Mutex or RwLock when you need to access the data from multiple threads.].

RAII

RAII stands for “resource acquisition is initialisation”, which is not a very helpful name; it’s a design pattern and means that every resource is managed by an object; when the object is created, the resource is acquired, and when the object is destroyed, the resource is released. This is what happens in the example above with the String: When s is created, a chunk of memory is acquired. When s is dropped, the memory is released again.

However, sometimes it’s useful to have a type that doesn’t acquire a resource immediately, and can still be used after the resource has been released. Also, sometimes we want to acquire and release a resource multiple times in a row. That’s where RAII guards come in.

RAII guards

RAII guards is a design pattern for managing a resource when RAII as described above is not flexible enough. It uses two types instead of one: The first one contains a resource (or has a way of acquiring it), but doesn’t allow us to access the resource directly. It does however have a method for creating an object of the second type. The second type is a RAII guard: It allows us to access the resource, until the RAII guard is dropped.

One example of the RAII guards pattern is RefCell. To mutate a RefCell, we have to call its borrow_mut method. This returns a write lock, which is a RAII guard:

use std::cell::RefCell;

fn main() {
    let ref_cell = RefCell::new(7);
    {
        // The RAII guard is created, which borrows the `RefCell` and acquires a write lock.
        let mut guard = ref_cell.borrow_mut();
        *guard *= 6;
    } // The RAII guard is dropped, which releases the write lock.
    dbg!(ref_cell);
} // The `RefCell` is dropped.

Implementation

Let’s implement a museum. People can come in and admire the exhibits:

pub struct Museum;

impl Museum {
    pub fn admire_exhibits(&self) {
        println!("How amazing!");
    }
}

Now because of the current pandemic, no more than 20 visitors are allowed inside at any time. To ensure this, the ticket seller has only 20 tickets. Visitors need a ticket to enter the museum, and when they exit, they have to give it back.

pub struct Museum {
    remaining_tickets: u32,
}

impl Museum {
    pub fn new() -> Self {
        Museum {
            remaining_tickets: 20,
        }
    }

    pub fn get_ticket(&mut self) -> Option<Ticket> {
        if self.remaining_tickets > 0 {
            self.remaining_tickets -= 1;
            Some(Ticket(()))
        } else {
            None
        }
    }

    pub fn return_ticket(&mut self, ticket: Ticket) {
        self.remaining_tickets += 1;
    }
}

pub struct Ticket(());

impl Ticket {
    pub fn admire_exhibits(&self) {
        println!("How amazing!");
    }
}

How great! We have a Museum and a Ticket type. Ticket has a private field, so it can’t be created directly from the public API. To get a ticket, one has to call Museum::get_ticket, which ensures that no more than 20 tickets are in circulation.

This design suffers from two problems, however: First, if there are multiple museums, we don’t know which ticket belongs to which museum. Although a visitor can’t teleport from one museum to another, Rust’s type system doesn’t know that, so it allows us to get a ticket from one museum and return it at a different one. This means that more visitors could be in the museum than intended, which is bad during a pandemic. This could be prevented if the tickets had the museum’s name on it, but that raises the question what to do when a wrong ticket is returned. We’d really like to avoid having to handle this type of error, if there’s an alternative.

The other problem is that this design doesn’t force visitors to return their ticket when exiting the museum: If we forget to call Museum::return_ticket, the ticket is lost forever. Once all tickets are lost, nobody will be able to enter the museum, even though it is empty!

Both problems can be solved by making Ticket a RAII guard, which borrows the museum:

use std::cell::RefCell;
use std::ops::Drop;

pub struct Museum {
    remaining_tickets: RefCell<u32>,
}

impl Museum {
    pub fn new() -> Self {
        Museum {
            remaining_tickets: RefCell::new(20),
        }
    }

    pub fn get_ticket(&self) -> Option<Ticket<'_>> {
        let mut lock = self.remaining_tickets.borrow_mut();
        if *lock > 0 {
            *lock -= 1;
            Some(Ticket { museum: self })
        } else {
            None
        }
    }
}

pub struct Ticket<'a> {
    museum: &'a Museum,
}

impl Ticket<'_> {
    pub fn admire_exhibits(&self) {
        println!("How amazing!");
    }
}

impl Drop for Ticket<'_> {
    fn drop(&mut self) {
        let mut lock = self.museum.remaining_tickets.borrow_mut();
        *lock += 1;
    }
}

Here’s what changed:

Ticket borrows the Museum, so it can’t be returned to the wrong museum.
Museum::get_ticket now takes &self instead of &mut self, because otherwise only one ticket could exist for each museum at a time.
remaining_tickets is wrapped in a RefCell, which provides interior mutability.
Museum::return_ticket was replaced with a Drop implementation for Ticket. This ensures that tickets are always returned.

Test it

As always, it is a good idea to test the code:

#[test]
fn test_museum() {
    let museum = Museum::new();
    let mut tickets: Vec<Ticket> = (0..20)
        .map(|_| museum.get_ticket().unwrap())
        .collect();

    assert!(museum.get_ticket().is_none());
    tickets.pop();
    tickets.push(museum.get_ticket().unwrap());
    assert_eq!(*museum.remaining_tickets.borrow(), 0);
    drop(tickets);
    assert_eq!(*museum.remaining_tickets.borrow(), 20);
}

Note that if the test is in the same module as the code, we can inspect private fields. Now let’s run the test:

> cargo test -q

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

When `Drop` isn’t called

Drop implementations (called destructors) are called whenever the scope of the destructor’s object is exited. It doesn’t matter how the scope is exited, e.g. it could be because of a return, continue or break statement, a ? expression or a panic. There’s one exception, however: When the process is exited, no destructors are run. This can be done e.g. by invoking abort, exit, or by panicking in a Drop implementation during another panic.

It is also possible to leak objects, which means that they will never be dropped. This is usually discouraged, but allowed. Objects can be leaked e.g. with the Box::leak function, or by creating a reference-counted graph with a cycle. Another risk are power outages and OS crashes: For obvious reasons, values aren’t dropped in such cases either.

So you can’t rely on destructors getting called. This is usually not a big problem, but something to be aware of when writing Drop implementations.

Fin

Discussion on Reddit. Please file a bug if you have questions, want some things explained in more detail, or if you found a mistake. See you around!

Rust's Universes

Wed, 10 Mar 2021 00:00:00 GMT

This post describes a curious feature of Rust: Namespaces, also called universes. Note that Rust’s namespaces are nothing like namespaces in languages such as TypeScript or C#; they are also unrelated to all of space and time, although there are certainly parallels.

Identifiers

To understand namespaces, we first need to talk about names. Namespaces contain all the names (called identifiers) in Rust code. This does not include keywords, but it includes the names of local variables, types, traits, functions, modules, generic arguments, macros, and so on.

Every identifier has a certain scope. For example, local variables are scoped to the block they were defined in, free functions are scoped to their module, trait methods are scoped to their trait, inherent methods to their type. Etcetera. It’s not possible to use an identifier outside of its scope, unless it was brought into scope with a use statement. So far so good.

But what happens when a scope contains multiple things with the same name? Now it gets complicated. In most cases, you’ll get a compiler error. An exception are local variables and macros, which use textual scoping, and therefore allow shadowing things with the same name:

let x = 5;
let x = "hello world!";  // no problemo!

macro_rules! x {
    ($e:expr) => {}
}
macro_rules! x {  // who cares!
    () => {}
}

These are the only exceptions, however. All other things use path-based scoping. While local variables and macros have to be declared before they can be used, things with path-based scoping can be declared and used in any order:

m!();  // error: m is not defined at this point
macro_rules! m {
    () => {}
}

f();  // this is fine
fn f() {}

Namespaces

However, path-scoped things have a limitation: There can’t be more than one thing with the same name defined in the same scope and namespace. For example:

fn wtf() {}
const wtf: u8 = 0;  // error: the name `wtf` is defined multiple times

mod wtf {}  // this is allowed‽

Why is that? the function wtf and the constant wtf both live in the same namespace, therefore their names clash. The module wtf however lives in a different namespace, so it can coexist with a function or constant of the same name. One might say, it’s in a parallel universe.

So you’re probably wondering, how many namespaces are there? Let’s look in the documentation of the rustc source! There are 3:

The type namespace
The value namespace
The macro namespace

Which items are part of which namespace is specified here in the DefKind type:

The type namespace contains
- modules
- structs
- enums
- unions
- enum variants
- traits
- type aliases
- foreign types
- trait aliases (currently unstable)
- associated types
- type parameters
The value namespace contains
- functions
- constants
- const parameters
- statics
- constructors
- associated functions
- associated constants
The macro namespace just contains macros.

All other names are treated specially and don’t fall into any of the above categories.

What does that mean?

Items with the same name can coexist in the same scope, if they are from different namespaces. One example for this are tuple structs, because they are desugared (i.e. transformed by the compiler) into a regular struct and a constructor function:

struct Foo(Bar);

// the compiler transforms the above into something like this:

struct Foo { 0: Bar };

fn Foo(_0: Bar) -> Foo {
    Foo { 0: _0 }
}

So this is why tuple structs can be both used as a type and invoked like a function. However, it also means that tuple structs occupy both the type namespace and the value namespace. Roughly the same happens with enum variants with round brackets. Furthermore, unit structs and unit-like enum variants expand to a type name and a value (a constant), so they also occupy both namespaces.

Resolving names from different namespaces

Rust’s syntax is designed to be unambiguous about the namespace in which the names live. It distinguishes between type positions and value positions, for example:

fn a(b: C) -> D {
    e::f::G.h::<I>(j)
}

Just by looking at the syntax, the compiler can tell that

C, D, e, f and I are in the type namespace
a, b, G, h and j are in the value namespace

How does that work? Let’s start with the obvious ones: a and h are functions, and b is a local variable, so they must be in the value namespace. j is used as a function argument, so it’s also a value. C and D are used in type positions, so they’re types. I is used as a generic argument, so it’s also a type.

That only leaves e, f and G. Since e and f are immediately followed by two colons (called the scoping operator), they must be in the type namespace. That makes sense, because types, traits and modules are in the type namespace. G however is followed by a dot, so it is parsed as a value.

Importing names

Items can be imported with a use item. But how are different namespaces handled? Generally, use imports items from all three namespaces. This means for example, when importing a tuple struct, both its type and its constructor are available.

There is an exception, however: When a path ends with ::{self} (the curly braces can contain more paths), only the name from the type namespace is imported. For example:

mod module {
    pub struct Foo();
}

// import both the type Foo and its constructor:
use module::Foo;

// import only the type:
use module::Foo::{self};

Are you confused yet?

End

I hope you enjoyed this post, even though it’s less practically useful than my previous post.

Discussion on Reddit.

If you have suggestions what topics I should cover next, please file a bug in the issue tracker. Also file a bug if you have questions or want some things explained in more detail, or found a mistake.

So long!

Creating an Iterator in Rust

Tue, 09 Mar 2021 00:00:00 GMT

When I woke up today, I thought, what a great day to start a blog! So here we are. Before we take off, just a short introduction: I’m Ludwig, I’m a CS student from Germany, and I love Rust. Since this blog is about Rust, I hope you do too!

This post is about a core concept in Rust, iterators. If you don’t know what iterators are, please read the chapter about iterators in the Rust book first.

The collection type

Iterators usually iterate over some sort of collection. Our collection type is a tree:

enum Node<Item> {
    Leaf(Item),
    Children(Vec<Node<Item>>),
}

Simple, right? A tree node is either a Leaf, in which case it contains an Item, or a list of child nodes.

A `traverse` method

We want to traverse (i.e. iterate over) this kind of tree depth-first. This means that when a node has multiple children, we first traverse the first child and all its descendants before moving on to the second child. This is easy to implement with a recursive algorithm:

impl<It> Node<It> {
    fn traverse(&self) {
        match self {
            Node::Leaf(_) => {}
            Node::Children(children) => {
                for node in children {
                    node.traverse();
                }
            }
        }
    }
}

If we want to do something with each item, we can pass a closure to the function:

impl<It> Node<It> {
    fn traverse(&self, f: impl Fn(&It)) {
        match self {
            Node::Leaf(item) => {
                f(item);
            }
            Node::Children(children) => {
                for node in children {
                    node.traverse(&f);
                }
            }
        }
    }
}

This allows us to iterate over the items. However, Node still doesn’t implement the Iterator trait, which would be useful because it provides helper methods such as filter, fold and collect.

As we will see shortly, implementing this trait is quite tricky in this case. The reason for this is that the Iterator trait provides external iteration, whereas our traverse method provides internal iteration.

Implementing the `Iterator` trait

To be able to externally iterate over the tree, we need to implement the Iterator trait. It looks roughly like this:

trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;
}

The iterator trait is usually not implemented for a collection directly. Instead, a new type is created that wraps the collection:

struct NodeIter<'a, It>(&'a Node<It>);

Why not implement Iterator for the collection?

This great question was asked recently on Reddit. One reason is, if the collection implemented the Iterator type directly, the collection could be modified during iteration, which is is error-prone and usually undesired. It would also make it impossible to have multiple iterators that iterate over the same collection simultaneously and independently from each other, because the Iterator::next() method borrows the iterator mutably.

More importantly, though, iterators usually need additional state to keep track of the position of the next item in the collection. This state needs to be updated in every iteration, which doesn’t work if the collection is behind an immutable reference and the state is part of the collection.

If the iterator is a separate type, it can be mutable even though the collection is immutable. You’ll see how this works in a moment, just read on!

As the immutable reference in the NodeIter type indicates, it can only iterate over immutable references of the items. To get an instance of this iterator, we add a .iter() method:

impl<It> Node<It> {
    fn iter(&self) -> NodeIter<'_, It> {
        NodeIter(self)
    }
}

Now we can start with the actual implementation!

impl<'a, It> Iterator for NodeIter<'a, It> {
    type Item = &'a It;

    fn next(&mut self) -> Option<Self::Item> {
        todo!()
    }
}

Now there’s a problem: Since Iterator provides external iteration, we have to produce one item at a time. This means that we can’t use the simple recursive algorithm we used in the traverse method. Instead, we have to keep track of the state of the iterator manually:

struct NodeIter<'a, It> {
    children: &'a [Node<It>],
    parent: Option<Box<NodeIter<'a, It>>>,
}

The children field contains the remaining children of a node, the parent field is the iterator of the parent node, if present. It must be wrapped in a Box because a struct in Rust can’t contain itself without indirection—otherwise, it would be impossible to compute its size on the stack.

So how does this work? When we create the iterator, we put the node into the children slice and set parent to None:

impl<It> Node<It> {
    fn iter(&self) -> NodeIter<'_, It> {
        NodeIter {
            children: std::slice::from_ref(self),
            parent: None,
        }
    }
}

When the iterator is advanced, we first check if children is empty. If that’s the case, we try to continue iterating the parent node. If there is no parent node, we return None.

If children is not empty, we remove the first child and check its variant. If it is a Node::Leaf, we return its content; if it is a Node::Children, we create a new iterator for the children. The parent field is set to self, and self is replaced with the newly created iterator:

use std::mem;

impl<'a, It> Iterator for NodeIter<'a, It> {
    type Item = &'a It;

    fn next(&mut self) -> Option<Self::Item> {
        match self.children.get(0) {
            None => match self.parent.take() {
                Some(parent) => {
                    // continue with the parent node
                    *self = *parent;
                    self.next()
                }
                None => None,
            },
            Some(Node::Leaf(item)) => {
                self.children = &self.children[1..];
                Some(item)
            }
            Some(Node::Children(children)) => {
                self.children = &self.children[1..];

                // start iterating the child trees
                *self = NodeIter {
                    children: children.as_slice(),
                    parent: Some(Box::new(mem::take(self))),
                };
                self.next()
            }
        }
    }
}

This doesn’t work yet, because mem::take() requires that NodeIter implements Default. But this can be fixed easily:

impl<It> Default for NodeIter<'_, It> {
    fn default() -> Self {
        NodeIter { children: &[], parent: None }
    }
}

Now let’s see if the iterator works!

Testing

To check if it works, we can write a unit test:

#[test]
fn test_borrowing_iterator() {
    let tree = Node::Children(vec![
        Node::Leaf(5),
        Node::Leaf(4),
        Node::Children(vec![
            Node::Leaf(3),
            Node::Leaf(2),
            Node::Children(vec![]),
        ]),
        Node::Children(vec![
            Node::Children(vec![
                Node::Children(vec![Node::Leaf(1)]),
                Node::Leaf(0),
            ]),
        ]),
    ]);

    let nums: Vec<i32> = tree.iter().copied().collect();
    assert_eq!(nums, vec![5, 4, 3, 2, 1, 0]);
}

> cargo test -q

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

That looks reassuring!

Adding features

Now that we have a working iterator, let’s see how we can improve it. First, let’s check if we can implement more iterator methods to make it more efficient!

Size hint

Every iterator has a size hint, to help the collect methods decide how much memory to allocate when collecting into something like a Vec. By default the lower bound of the size hint is 0, so the collect method might have to re-allocate a few times. This is still better than setting the size hint too high, because that would waste memory.

Unfortunately, we don’t know how many elements a Node contains, and calculating the number of elements would be expensive, so we’ll skip the size_hint method.

`FusedIterator`

Sometimes it’s useful to ensure that after the iterator produces None for the first time, it will only produce None values. Iterators with this property are called fused iterators, and any iterator can be converted to a fused iterator with the .fused() method.

However, if we implement the FusedIterator trait for our iterator, calling the .fused() method is more efficient, because it has a specialized implementation for types that implement this trait. So let’s add it:

use std::iter::FusedIterator;

impl<It> FusedIterator for NodeIter<'_, It> {}

That’s it!

`IntoIterator`

This trait doesn’t make the iterator more efficient, just more ergonomic. Implementing IntoIterator for &Node<T> makes it possible to use a node in a for loop without having to write .iter() explicitly:

impl<'a, It> IntoIterator for &'a Node<It> {
    type Item = &'a It;

    type IntoIter = NodeIter<'a, It>;

    fn into_iter(self) -> Self::IntoIter {
        self.iter()
    }
}

Let’s try it out:

#[test]
fn test_borrowing_for_loop() {
    let tree = Node::Leaf(42);

    for &node in &tree {
        let _: i32 = node;
    }
}

And… it compiles! 🎉

An owned iterator

We can also implement an iterator that consumes the tree and produces the items as owned values. To implement this iterator, we can copy-paste the borrowed iterator and make a few adjustments:

struct NodeIntoIter<It> {
    // we use a VecDeque because it allows
    // removing elements from the front efficiently pass
    children: VecDeque<Node<It>>,
    parent: Option<Box<NodeIntoIter<It>>>,
}

impl<It> Default for NodeIntoIter<It> {
    fn default() -> Self {
        NodeIntoIter {
            children: Default::default(),
            parent: None,
        }
    }
}

impl<It> Iterator for NodeIntoIter<It> {
    type Item = It;

    fn next(&mut self) -> Option<Self::Item> {
        match self.children.pop_front() {
            None => match self.parent.take() {
                Some(parent) => {
                    // continue with the parent node
                    *self = *parent;
                    self.next()
                }
                None => None,
            },
            Some(Node::Leaf(item)) => Some(item),
            Some(Node::Children(children)) => {
                // start iterating the child trees
                *self = NodeIntoIter {
                    children: children.into(),
                    parent: Some(Box::new(mem::take(self))),
                };
                self.next()
            }
        }
    }
}

Now let’s implement IntoIterator for Node, so we can use it:

impl<It> IntoIterator for Node<It> {
    type Item = It;

    type IntoIter = NodeIntoIter<It>;

    fn into_iter(self) -> Self::IntoIter {
        let mut children = VecDeque::with_capacity(1);
        children.push_back(self);

        NodeIntoIter {
            children,
            parent: None,
        }
    }
}

Don’t forget to test it:

> cargo test -q

running 4 tests
....
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

A mutable iterator

Collections usually also have an iterator for mutating the items. They can be particularly tricky to implement safely, because you have to ensure that no part of the iterator is ever borrowed mutably multiple times.

But because this blog post is already way too long, I leave this part as an exercise to the reader. 😛

If you got stuck implementing this yourself, you may take a peek at this playground.

What about `DoubleEndedIterator`?

DoubleEndedIterator is a trait implemented by iterators that can consume items from both ends. However, we won’t implement this trait for our iterators, because it would make them much more complicated. And who needs that trait anyway? 😉

Fin

You should now be able to implement iterators for tree-like data structures.

All code is available in this playground. Discussion on Reddit.

If you have suggestions what topics I should cover next, please file a bug in the issue tracker. Also file a bug if you have questions or want some things explained in more detail, or if you found a mistake.

I will write posts regularly from now on. If you enjoyed this post, please subscribe to the atom feed (see at the bottom) and share it with your friends! Until next time!

Aloso's blog | Aloso's Blog

Template strings in Rust

Motivation

Why shouldn’t we do this?

The idea

A new approach

The Pretty trait

Supporting no_std

Effects in Rust (and Koka)

Side effects in Rust

Effects that modify control flow

Panics (exceptions)

Await and yield

Effect Polymorphism

Keyword Generics

Constness

Encoding side effects in the type system

Conclusion

We don't need revenge

Background

Accountability, whatever that means

The ethics of punishment

Why do we need to know whose fault it was?

A zero-overhead linked list in Rust

Implementation

Persistent data structures

Writing an iterator

Internal iteration

Limitations

Use case

Alternatives

Fin

Rusts Module System Explained

Why do modules exist?

The module tree

Submodules

An example

Items and paths

Visibility

Encapsulation

Fine-grained visibility

Visibilities overview

Exports

Common pitfalls

Special cases

Enum variants and fields

Macros

Why are modules declared explicitly?

Fin

Implementing RAII guards in Rust

Where is Drop being used?

RAII

RAII guards

Implementation

Test it

When Drop isn’t called

Fin

Rust's Universes

Identifiers

Namespaces

What does that mean?

Resolving names from different namespaces

Importing names

End

Creating an Iterator in Rust

The collection type

A traverse method

Implementing the Iterator trait

Testing

Adding features

Size hint

FusedIterator

IntoIterator

An owned iterator

A mutable iterator

What about DoubleEndedIterator?

Fin

The `Pretty` trait

Supporting `no_std`

Where is `Drop` being used?

When `Drop` isn’t called

A `traverse` method

Implementing the `Iterator` trait

`FusedIterator`

`IntoIterator`

What about `DoubleEndedIterator`?