Adding support for receiving webmentions seems like a good place to start. This site is a static site built with Jekyll and I want it to stay that way. Since the standard allows for the receiver endpoint to be a different server, I thought I’d make a separate application to run on my homeserver. I have a private repo for the site on my Forgejo instance so it would need to make commits and pull requests there which I could then merge to trigger CI to update the site.

Rust is my go-to language nowadays and seems like a reasonable fit for this kind of project. The applications consists of a single endpoint HTTP server which handles incoming requests, parses them, does some basic validation and then queues them for processing by a worker task which does more in-depth validation and communicates with Forgejo.

Altogether, it’s less than 1,000 lines of code and less than a day of work to get something that I’m happy with. The whole thing is built into a docker container and is now running happily on my homeserver. If you’d like to checkout the code you can find the repo at https://forgejo.zacjw.com/zac/webmention-receiver

Next step will be actually integrating it into the site so the files it creates become links at the end of posts. I’ll also add a little form into each post for adding a webmention in case your site doesn’t automatically send webmentions.

You can find my fork with this feature at https://forgejo.zacjw.com/zac/jekyll-feed

Use of the new feature is pretty simple. Here’s an excerpt of this site’s _config.yml:

feed:
  combined_feeds:
    all_posts:
      collections:
        blog:
        projects:
          except:
            is_top_level: true

This also shows off that I added a filtering feature to the combined feeds since I needed a way to exclude certain project posts from the feed. I’ve not tried upstreaming this yet (I feel like the filtering feature is pretty niche so they might not accept it), but maybe I’ll do that at some point.

History of this website

This website wasn’t always written in Jekyll. When I started this site I didn’t even know about static site generators. The thought of ‘compiling’ a HTML page didn’t cross my mind. I assumed the only option besides manually authoring each page was to do something completely dynamic. It’s for this reason that I started this site on top of Wordpress.

Wordpress was functional. I liked being able to write posts in the browser, but it was never a must have feature for me. Eventually I got fed up with the continuous stream of security patches that had to be applied and decided to rebuild the site.

At this point I still didn’t know about static site generators so I decided to build my only dynamic site generator. It was a few PHP scripts and a few snippets of HTML. It didn’t really work well. I never even tried making an in browser editor for posts. Turns out my attempt was just as insecure as Wordpress, so I gave up on it after a while.

The I discovered GitHub Pages and Jekyll. It was eye-opening. I hadn’t considered adding a build step and then just serving static files. I threw away the existing themes and jumped right into custom layouts to make sites that worked how I wanted. I eventually applied what I had learnt to this site, though I never did write very much for it.

Recently I had a look at Amos Wenger’s dodeca static site generator. He generally does really good work so I was expecting it to be pretty smooth to build a new site with it. It was anything but. The templating engine seems completely unable to express basic conditionals like ‘if the [extra] section of this post’s front matter has a particular key defined, render this section’. Instead I would get errors telling me the key wasn’t defined in the very condition where I was trying to check if it was. I promptly gave up and decided to build a new site but still in Jekyll.

This time I did more work with custom plugins to let me group certain posts into projects and have a nice navigation experience. I also spent more effort on how the site looks but that could have been done in any competent SSG.

My Priorities for a Static Site Generator

Beyond the fundamentals of a permissive templating engine, my single highest priority is plugins. Plugins make it possible to customise the SSG beyond what can be done within the templating language. I’ve already mentioned how I use a custom plugin to support grouping posts into projects, but I also use it for the recommendations at the end of blog posts. Each post has a list of tags in its front matter, and it just looks for other posts with lots of tags in common. I couldn’t come up with a way to do it in liquid, but it was all of about 30 lines of Ruby.

Dodeca doesn’t have plugins (at least not at the time of writing). It does have ‘Build Steps’ but, as far as I can tell, they’re just a nice way to hook a command to be run at build time and reference the output from your templates. It can’t transform posts between parsing and rendering, or inject synthetic posts.

I did also look at Cobalt after the bad experience with Dodeca, but it too doesn’t offer a plugin system, and at this point I consider one a need-to-have.

Building a static site generator where the plugin system is the headline feature

What might a static site generator look like if it were built around its plugin system. I think it ought to be fairly agnostic over the language used for the plugins. I managed to use Ruby for the plugins I needed to write for Jekyll, but it was complicated by me never having used the language before. I don’t have any interest in Ruby as a language so if Jekyll gave me more options I probably would have chosen something else.

If plugins are a first-class feature, it should be easy to bring in 3rd party plugins. Part of making it easy is not having to worry about malware pretending to be just the plugin you need. So plugins should be isolated from the host so they can’t steal all of your browser’s cookies and saved passwords or encrypt and ransom your files. Isolating the plugins means a plugin will have to bring its dependencies with it, but that’s probably for the best since it will improve compatibility.

On the topic of compatibility, ARM CPUs are becoming more common (and RISC V might get there eventually too) so plugins should be agnostic of host CPU architecture.

This all points to running the plugins in some kind of thin architecture-agnostic virtual machine that can be targeted by many languages. Web Assembly is a thin architecture-agnostic virtual machine that can be targeted by many languages. I’m hardly the first to imagine using Web Assembly for this purpose; the Extism project looks perfect for this. Looking more closely at if it can handle fine-grained isolation, I’m not sure it can so I can’t just use it as-is but I can at least use it as a guide.

A good way to make sure that the plugin system is sufficiently powerful might be to actually make every part of it beyond the plugin runtime itself a plugin. It’ll end up being very modular, like a build-your-own-SSG toolkit.

The rest of this project will document building such a static site generator and hopefully by the end of it I’ll have a tool worth rewriting my site in yet again.

Implementation

Marshall

I’ll introduce the Marshall implementation first since it’s quite short but does have some details worth pointing out.

/// An executor marshall that is only safe to be used from a single thread.
///
/// Its implementation of [Sync] is only so that you can have static values.
pub struct SingleThreadMarshall {
    ptr: Cell<*const Cell<bool>>,
}

// SAFETY: creating values of this type is unsafe and requires upholding a
// contract that ensures no UB can occur due to what would otherwise be an
// unsound impl of Sync for this type.
unsafe impl Sync for SingleThreadMarshall {}

impl SingleThreadMarshall {
    /// You must not share references to this value between threads. All method calls
    /// on this value must come from the same thread for its entire lifetime.
    ///
    /// If this was called outside of a static initialiser, the value can only be used
    /// from the thread that created it.
    ///
    /// Its implementation of [Sync] is only so that you can have static values.
    pub const unsafe fn new() -> Self {
        Self {
            ptr: Cell::new(core::ptr::null()),
        }
    }

    pub unsafe fn register(&self, b: &Cell<bool>) {
        self.ptr.set(b);
    }

    pub unsafe fn unregister(&self) {
        self.ptr.set(core::ptr::null());
    }
}

Unfortunately this implementation ends up with some unsafe and worse, a dangerous unsafe Sync impl for a type that clearly can’t be shared between threads. This is an example of Rust’s assumptions about the target environment being wrong and then subsequently constraining us more tightly than is actually necessary for our target.

In Rust, static variables must be of types that implement Sync since the language can’t (or at least doesn’t) restrict access by thread. That lets us ‘fearlessly’ add multithreading to our programs without risk of UB, but it’s not a free lunch. Our types have to be designed to actually be thread-safe and that’s on us. Since my goal is to test on an AtMega328, the Sync requirement is all downside, no upside as there’s no threading on that target anyway.

You may wonder why I don’t use thread_local! since that lets you avoid the Sync requirement. It’s because it’s not part of core, it’s part of std, and goal 1 was to support no-std targets.

This boxes me into the unsafe impl with little alternative. To try to make it a bit clearer how dangerous this type is for threaded targets, I’ve made the new method unsafe with a doc comment that describes a contract that undoes the Sync.

With that annoying detail out of the way, we can talk about why register and unregister are also unsafe. The marshall’s job is to provide a 'static lifetime stepping stone for a waker to send a signal to a task manager. The task manager might have a lifetime shorter than 'static so the only way to keep the marshall 'static is to unsafely extend the reference lifetime or equivalently use a raw pointer. The task manager could be destroyed or moved while a waker tied to this marshall still exists, which in the &'static case would be instant UB and in the raw pointer case would be UB when the waker attempts to signal the task manager. Making registering a task manager with the marshall unsafe makes it clear that it is the programmers responsibility to unregister before moving or destroying the task manager.

This probably could be simplified with some sort of registration handle which borrows the task manager and unregisters it from the marshall when the handle is dropped, but it would probably have to involve pinning to guarantee that it is actually dropped and not forgotten. It might also cause issues with not being able to mutably borrow the task manager for anything else, so for now this manual and more error-prone approach will have to do.

The implementation of the Marshall trait is straightforward:

impl Marshall for SingleThreadMarshall {
    fn wake(&'static self) {
        let ptr = self.ptr.get();
        if ptr.is_null() {
            return;
        }
        unsafe {
            (&*ptr).set(true);
        }
    }

    fn waker(&'static self) -> core::task::Waker {
        unsafe {
            core::task::Waker::from_raw(core::task::RawWaker::new(
                self as *const Self as *const (),
                &SINGLE_THREAD_MARSHALL_VTABLE,
            ))
        }
    }
}

The SINGLE_THREAD_MARSHALL_VTABLE just calls the marshall’s wake method for both wake and wake_by_ref.

Implementing FusedFutureWithWakeStatus

In the previous section I talked about how the marshall signals the task manager, but that isn’t really true. It signals the task directly as it is the task which must hold its wake status (that’s what FusedFutureWithWakeStatus does). It’s worth introducing an implementation of FusedFutureWithWakeStatus to make that more clear.

#[pin_project::pin_project]
pub struct SingleThreadWithWakeStatus<F> {
    #[pin]
    f: F,
    status: core::cell::Cell<bool>,
}

impl<F> SingleThreadWithWakeStatus<F> {
    pub unsafe fn register(self: Pin<&mut Self>, marshall: &'static SingleThreadMarshall) {
        let projection = self.project();
        marshall.register(&projection.status);
    }
}

Notice that this implementation is specifically tied to SingleThreadMarshall as they both agree on using a Cell<bool> for storing the wake status.

We then have to implement the future traits for this type:

impl<F: Future> Future for SingleThreadWithWakeStatus<F> {
    type Output = F::Output;

    fn poll(
        self: Pin<&mut Self>,
        cx: &mut core::task::Context<'_>,
    ) -> core::task::Poll<Self::Output> {
        let projection = self.project();
        projection.f.poll(cx)
    }
}

impl<F: FusedFuture> FusedFuture for SingleThreadWithWakeStatus<F> {
    fn is_terminated(&self) -> bool {
        self.f.is_terminated()
    }
}

impl<F: FusedFuture> FusedFutureWithWakeStatus for SingleThreadWithWakeStatus<F> {
    fn status(&self) -> WakeStatus {
        if self.status.get() {
            WakeStatus::Woken
        } else {
            WakeStatus::Asleep
        }
    }

    fn set_status(self: Pin<&mut Self>, status: WakeStatus) {
        match status {
            WakeStatus::Woken => self.status.set(true),
            WakeStatus::Asleep => self.status.set(false),
        }
    }
}

Finally an extension trait can make it a bit more ergonomic to set tasks up:

pub trait FusedFutureExt {
    fn with_wake_status_st(self) -> SingleThreadWithWakeStatus<Self>
    where
        Self: Sized;
}

impl<F: futures::future::FusedFuture> FusedFutureExt for F {
    fn with_wake_status_st(self) -> SingleThreadWithWakeStatus<Self> {
        SingleThreadWithWakeStatus {
            f: self,
            status: core::cell::Cell::new(true),
        }
    }
}

Task Manager

pub struct ArrayTaskManager<'a, const N: usize, MarshallType: Marshall> {
    pub tasks: [(
        Pin<&'a mut dyn FusedFutureWithWakeStatus<Output = ()>>,
        &'static MarshallType,
    ); N],
}

Here’s a simple task manager type. It doesn’t support spawning tasks since it has a const generic fixed capacity. It also doesn’t own the tasks, but instead holds pinned mutable references to them. This makes everything a consistent size, without which we wouldn’t be able to use an array in this way. Each task must have its own marshall so tasks is an array of task-marshall pairs.

Implementing the TaskManager trait is as follows:

impl<'a, const N: usize, Marshall: ExecutorMarshall> TaskManager
    for ArrayTaskManager<'a, N, Marshall>
{
    type Marshall = Marshall;

    type TaskIterator<'b> = ArrayTaskManagerIter<'b, 'a, N, Marshall>
    where
        Self: 'b;

    fn get_task(
        &mut self,
        i: usize,
    ) -> Option<(
        Pin<&mut dyn FusedFutureWithWakeStatus<Output = ()>>,
        &'static Self::Marshall,
    )> {
        self.tasks
            .get_mut(i)
            .map(|(task, marshall)| (unsafe { core::mem::transmute(task.as_mut()) }, *marshall))
    }

    fn sleep_task(&mut self, i: usize) {
        self.tasks[i]
            .0
            .as_mut()
            .set_status(crate::WakeStatus::Asleep);
    }

    fn sleep_all(&mut self) {
        for task in &mut self.tasks {
            task.0.as_mut().set_status(crate::WakeStatus::Asleep);
        }
    }

    fn tasks<'b>(&'b mut self) -> Self::TaskIterator<'b> {
        ArrayTaskManagerIter {
            array: &mut self.tasks,
            i: 0,
        }
    }
}

I was running into all kinds of variance problems when writing this (the ArrayTaskManagerIter implementation which I’ve omitted has more too) but besides that the implementation is unsurprising.

Testing

To test it out I want to build a simple program that blinks an LED and prints over UART periodically, but those periods should be different.

Task Definitions

async fn blink_task(mut pin: impl OutputPin) {
    loop {
        let _ = pin.set_high();
        sleep(1000).await;
        let _ = pin.set_low();
        sleep(1000).await;
    }
}

async fn print_task() {
    for i in 0.. {
        println!("i={i}");
        sleep(2500).await;
    }
}

These tasks are very simple but should serve as a good first test, but before I can try it I need to implement the sleep future, as well as println! since that normally comes from std.

sleep

I took an implementation of a Rust version of Arduino’s millis from avr-hal to build this future around it:

#[pin_project::pin_project]
pub struct Sleep {
    end: u32
}

impl Future for Sleep {
    type Output = ();

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        let projection = self.project();
        let now = millis();
        if now.wrapping_sub(*projection.end) <= (*projection.end).wrapping_sub(now) {
            cx.waker().wake_by_ref();
            Poll::Ready(())
        } else {
            cx.waker().wake_by_ref();
            Poll::Pending
        }
    }
}

pub fn sleep(sleep_millis: u32) -> Sleep {
    Sleep { end: millis() + sleep_millis }
}

Because we are operating without an event pool the future has no way to signal to the executor what it is actually waiting for so it opts to request wake up with cx.waker().wake_by_ref() then return Poll::Pending. This will cause the executor to repeatedly poll this task until sufficient time has passed, but it will interleave polling the other tasks so not to effective block as that would defeat the purpose. This is a wasteful way to write a sleep future, but should still work.

println!

Though not on the surface of particular relevance to building an executor, I though I’d include how I implemented println! as there are some interesting details.

pub struct GlobalSerial {
    serial: UnsafeCell<
        Option<
            avr_hal_generic::usart::Usart<
                Atmega,
                USART0,
                avr_hal_generic::port::Pin<Input, PD0>,
                avr_hal_generic::port::Pin<Output, PD1>,
                MHz16,
            >,
        >,
    >,
}

unsafe impl Sync for GlobalSerial {}

impl GlobalSerial {
    pub unsafe fn init(
        &self,
        usart: avr_device::atmega328p::USART0,
        rx: avr_hal_generic::port::Pin<Input, PD0>,
        tx: avr_hal_generic::port::Pin<Output, PD1>,
        baudrate: u32,
    ) {
        unsafe {
            *self.serial.get() = Some(avr_hal_generic::usart::Usart::new(
                usart,
                rx,
                tx,
                baudrate.into_baudrate::<MHz16>(),
            ))
        };
    }
}

impl core::fmt::Write for &'static GlobalSerial {
    fn write_str(&mut self, s: &str) -> core::fmt::Result {
        avr_device::interrupt::free(|_| -> core::fmt::Result {
            let Some(serial) = (unsafe { &mut *self.serial.get() }) else {
                return Err(core::fmt::Error::default());
            };
            for byte in s.as_bytes() {
                nb::block!(serial.write(*byte)).map_err(|_| Default::default())?
            }
            Ok(())
        })
    }
}

pub static SERIAL: GlobalSerial = GlobalSerial {
    serial: UnsafeCell::new(None),
};

That annoying unsafe impl Sync appears again. avr_device::interrupt::free is used to prevent UB in the event that an interrupt handler tries to print (if we were interrupted during a print we may have already acquired a mutable reference to the USART through the UnsafeCell but our print in the interrupt handler would then acquire the mutable reference again while the other reference still exists which is UB). The part of this implementation I like the least is probably the use of nb::block!, but its necessity highlights some issues with Rust’s fmt module.

Ideally I would like to write asynchronously since writing over serial can take a long time, especially at low baudrates. One approach might be to define an asynchronous write trait but we have no way to pass such an implementor to any of the formatting traits like Display since they take a Formatter<'a> which encapsulates a &'a dyn core::fmt::Write among other things. We could replace the formatting traits too but then we encounter another issue, core::fmt::Arguments<'a> is almost completely opaque. We’d have to not only replace the write trait and the formatting traits, but format_args! as well. That’s not really practical for an initial test so for now we’ll have to settle for a synchronous blocking implementation and so nb::block! is here to stay.

The println! macro is then defined in terms of writeln! with the static SERIAL writer.

#[macro_export]
macro_rules! println {
    ($format_string:literal $(, $($name:ident = )? $e:expr)* $(,)?) => {
        writeln!(&crate::serial::SERIAL, $format_string $(, $($name = )? $e)*).unwrap()
    };
}

main

Here’s the rest of the code so you can see how the executor is setup:

static SLEEP_TASK_MARSHALL: SingleThreadMarshall = unsafe { SingleThreadMarshall::new() };

static PRINT_TASK_MARSHALL: SingleThreadMarshall = unsafe { SingleThreadMarshall::new() };

#[arduino_hal::entry]
fn main() -> ! {
    let peripherals = Peripherals::take().expect("Failed to get peripherals");
    let pins = arduino_hal::pins!(peripherals);
    millis_init(peripherals.TC0);
    unsafe {
        SERIAL.init(
            peripherals.USART0,
            pins.d0.into_floating_input().forget_imode(),
            pins.d1.into_output(),
            115200,
        )
    };
    let mut blink_task = core::pin::pin!(blink_task(pins.d13.into_output())
        .fuse()
        .with_wake_status_st());
    unsafe { blink_task.as_mut().register(&SLEEP_TASK_MARSHALL) };
    let mut print_task = core::pin::pin!(print_task().fuse().with_wake_status_st());
    unsafe { print_task.as_mut().register(&PRINT_TASK_MARSHALL) };

    let mut tasks = ArrayTaskManager {
        tasks: [
            (blink_task, &SLEEP_TASK_MARSHALL),
            (print_task, &PRINT_TASK_MARSHALL),
        ],
    };
    let mut pool = DummyPool;

    let executor = Executor::new(&mut tasks, &mut pool);

    println!("Starting Execution");
    unsafe { avr_device::interrupt::enable() };
    executor.run_to_completion();
    println!("Execution Complete");

    loop {}
}

And… it works! The tasks blink the onboard LED and periodically prints over UART. I’ve omitted the implementation of DummyPool since it’s not interesting as it doesn’t do anything. This has proved to me that async Rust can be used even on tiny embedded systems like the AtMega328.

Next Steps

In the next part I’ll start to build the event source pool and try using it to improve the sleep future and to add a task that waits for a button to be pressed before doing something else.

Goals

Goal 1: My goal isn’t to build a better Tokio, that seems like setting myself up for failure. Instead my goal is to do what Tokio can’t do, run on small embedded systems. I could target the ESP32 since I work with them a lot, but since you can build Rust’s std on top of esp-idf I wouldn’t be surprised if it could just run Tokio anyway. Instead I’ll do my testing on something much smaller, an Arduino Nano. There’s no std support for AVR targets since they’re much too small, and since Tokio normally runs on machines with at least tens of gigabytes of RAM, it’s probably more optimised for that kind of use so I doubt it could fit into the 2.5 kilobytes of RAM of the AtMega328.

Goal 2: Beyond targeting tiny embedded systems without std, I also want to support no-alloc environments too. With so little memory to work with on these systems, the predictability of static and stack only are especially valuable. I want to offer extra features if alloc is available, but it shouldn’t be mandatory.

Goal 3: Finally, although I’ll be testing on an Arduino Nano, the whole thing should be agnostic of what it’s running on. All of the target specific logic should be able to be injected into it as needed by the consuming application.

Design

The way I see it an executor needs a few main parts:

• Something to hold spawned tasks without moving them around since they need to be pinned.
• Some kind of event loop where the executor will do IO operations on behalf of the futures running in the executor
• Something to facilitate communication between these two pieces when a task needs to be woken up.

The executor is then just going to be a thin wrapper over these parts. Each of them will get a trait and the executor will take in implementors of these traits in its new method. That way the user can customise aspects of the executor’s function by implementing them on their own types.

The event loop part doesn’t seem like it should be strictly necessary so I’ll leave it out of my proof-of-concept and work on its design later.

Task Manager

Initial requirements for this part are to have some way to iterate over the stored tasks so the executor can poll them, and also to have a way to put tasks to sleep. The executor can’t hold the wake status of each task since it doesn’t know how many there are and would need to allocate at runtime to do so. Goal 2 precludes this requirement so that responsibility has to be pushed onto the implementor of this trait.

Here’s what that trait might look like:

pub trait TaskManager {
    type TaskIterator<'a>: Iterator<
        Item = Pin<&'a mut dyn Future<Output = ()>>
    >
    where
        Self: 'a;
    fn get_task(
        &mut self,
        i: usize,
    ) -> Option<Pin<&mut dyn Future<Output = ()>>>;
    fn sleep_task(&mut self, i: usize);
    fn sleep_all(&mut self);
    fn tasks<'a>(&'a mut self) -> Self::TaskIterator<'a>;
}

I didn’t realise this until after I had written it, but here’s a great use for Rust’s recent MVP support for generic associated types (GATs).

The only thing the jumps out as needing improvement here is the i: usize in get_task and sleep_task. The idea I had was that it should correspond with when in the iterator sequence the task gets returned, but that probably imposes too many restrictions on the implementor. I’ll continue with this for now but it’ll need some more work later.

Context

Looking at the Future trait…

pub trait Future {
    type Output;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

…the next thing to figure out is Context<'a> since I’ll need those to be able to poll the tasks. Initially it looks quite good. There’s a lifetime parameter on it so maybe this communication part mentioned earlier will be able to exist on the stack. I’ve looked at this type before and remember that there’s a RawWaker which is sort of like a &dyn WakerTrait except WakerTrait doesn’t exist and instead you have to make the V-table manually. That RawWaker then gets put behind a wrapper, Waker for some reason that I don’t understand.

Updated: The Wake trait does actually exist, but only in alloc and std, not in core. It requires the use of Arc which makes it incompatible with core.

Looking at Waker I notice that unlike Context<'a>, it doesn’t have a lifetime parameter. I guess if you only ever have a 'a reference to it from a context it’s basically the same thing but then I notice it implements Clone. So much for having the communication part be able to live on the stack, with a Waker that can live forever so too must anything it references also live forever. This means I need 'static lifetimes for the Waker’s references which means static or heap and since I don’t want to require alloc, I’ll have to keep in mind as I design that the communication part will be a static variable. Let’s give this communication part the name…

Marshall

I think I’ve seen this term used in the context of interrupt handlers where you want to go from a function pointer to making a method call on a particular object. This situation of having to go from a 'static to a 'a feels similar to me so I’ll use the same name for it.

Here’s the trait:

pub trait Marshall: Sync + 'static {
    fn wake(&'static self);
    fn waker(&'static self) -> core::task::Waker;
}

The documentation for Waker makes note that it implements Send and Sync. Since I think my waker will basically be a reference to an implementor of Marshall, that implementor will need to be Sync to be safe in multi-threaded environments. Although maybe not strictly necessary, I’ve included a wake method in addition to the waker since it was easier for me to think about it this way.

This trait is kept deliberately very simple, nothing about how it communicates with a task manager is present. This is because I expect the TaskManager implementor and Marshall implementor to be written as a compatible pair and it makes more sense to have the bound be on the TaskManager side, which now becomes:

pub trait TaskManager {
    type Marshall: Marshall;
    type TaskIterator<'a>: Iterator<
        Item = (
            Pin<&'a mut dyn Future<Output = ()>>,
            &'static Self::Marshall,
        ),
    >
    where
        Self: 'a;
    fn get_task(
        &mut self,
        i: usize,
    ) -> Option<(
        Pin<&mut dyn Future<Output = ()>>,
        &'static Self::Marshall,
    )>;
    fn sleep_task(&mut self, i: usize);
    fn sleep_all(&mut self);
    fn tasks<'a>(&'a mut self) -> Self::TaskIterator<'a>;
}

The executor will need to be able to get a reference to the Marshall to be able to construct a Context<'a> to poll with, and depending on implementation each task may need to have it’s own Marshall instance so making it part of the task iterator and getter seems right.

Executor

Now seems like a good point to start writing the Executor wrapper type. Here’s the first version:

pub struct Executor<'a, Tasks: TaskManager> {
    tasks: &'a mut Tasks,
}

For now it’s just a wrapper over a mutable reference to a TaskManager but I expect there to be more in here once the event loop part is built.

Methods on the type look like this:

impl<'a, Tasks: TaskManager> Executor<'a, Tasks> {
    pub fn new(tasks: &'a mut Tasks) -> Self {
        Executor { tasks }
    }

    pub fn run_to_completion(self) {
        while self.tasks.tasks().any(|(task, _)| !task.is_terminated()) {
            for (i, (mut task, marshall)) in
                self.tasks.tasks().enumerate().filter(|(_, (task, _))| {
                    todo!("What goes here?")
                })
            {
                let waker = marshall.waker();
                let mut context: core::task::Context<'_> = core::task::Context::from_waker(&waker);
                self.tasks.sleep_task(i);
                let _ = task.poll(&mut context);
            }
        }
    }
}

Unfortunately, run_to_completion doesn’t compile. We’re borrowing from self.tasks for as long as we have a pinned reference to the task, so we can’t then borrow again to set the sleep status. Also, I want to be able to filter out tasks that are asleep or completed since they don’t need to be polled, but I don’t have the means to check for either. Luckily both of these issues can be solved with the same technique, use a sub-trait of Future for the tasks.

If we depend on the futures we can use its FusedFuture trait so we can check if a task is complete. Similarly we can define our own sub-trait of FusedFuture which stores a wake status too.

pub trait FusedFutureWithWakeStatus: FusedFuture {
    fn status(&self) -> WakeStatus;
    fn set_status(self: Pin<&mut Self>, status: WakeStatus);
}

WakeStatus is just a two-state enum:

pub enum WakeStatus {
    Woken,
    Asleep,
}

set_status needs to be callable when pinned so Pin appears in the self type. Since the status method is immutable it’s not needed.

TaskManager now becomes:

pub trait TaskManager {
    type Marshall: Marshall;
    type TaskIterator<'a>: Iterator<
        Item = (
            Pin<&'a mut dyn FusedFutureWithWakeStatus<Output = ()>>,
            &'static Self::Marshall,
        ),
    >
    where
        Self: 'a;
    fn get_task(
        &mut self,
        i: usize,
    ) -> Option<(
        Pin<&mut dyn FusedFutureWithWakeStatus<Output = ()>>,
        &'static Self::Marshall,
    )>;
    fn sleep_task(&mut self, i: usize);
    fn sleep_all(&mut self);
    fn tasks<'a>(&'a mut self) -> Self::TaskIterator<'a>;
}

And Executor::run_to_completion becomes:

pub fn run_to_completion(self) {
    while self.tasks.tasks().any(|(task, _)| !task.is_terminated()) {
        for (i, (mut task, marshall)) in
            self.tasks.tasks().enumerate().filter(|(_, (task, _))| {
                !task.is_terminated() && matches!(task.status(), WakeStatus::Woken)
            })
        {
            let waker = marshall.waker();
            let mut context: core::task::Context<'_> = core::task::Context::from_waker(&waker);
            task.as_mut().set_status(WakeStatus::Asleep);
            let _ = task.poll(&mut context);
        }
    }
}

Since we’re depending on FusedFuture we can just discard the result of task.poll as it’s implementation will update termination status for us if needed.

Conclusion

That’s it for the first part in this series. We’ve laid down the core ideas and formalised them into some traits. In the next part I’ll be implementing these traits to see how they work in practise.

It’s still early days for this project, but here’s a quick rundown of the features it has at time of writing:

Piping

Piping is supported with .pipe() after a call to run. You can chain as many of them as you would like.

from shell_scripter import run
run("dmesg").pipe("grep", "usb")().show()

Delayed Execution

You may be wondering what the ().show() is doing on the example above. Shell-scripter doesn’t run anything until you call the command object. This allows you to build up chains bit by bit and then execute when everything is ready. .show() just prints the output of the command to stdout.

Background Execution

If you want a command to execute without blocking the execution of your Python script, call the command object with (False) or (wait=False).

Prerequisites

• A working knowledge of git and SSH
• A server to be the remote git repo and to perform the Jekyll build
• If that server is not where you want to host the website, you’ll need some other hosting

How to setup a git remote

You don’t need to install anything on your server beyond the normal git CLI and an SSH server. On Debian/Ubuntu that’s as simple as sudo apt-get install git openssh-server

Next you need to decide which user to use for the git operations. This will also be the user that the deploy script is executed as. I’ll be creating a user called git. That can be done with sudo adduser git. This makes a new user and gives them a home directory, in this case /home/git/ which we can use to keep our repos in.

(Optional) SSH key authentication

If you want to use key authentication rather than password authentication, we need to run sudo su git to get a shell as our new user, then mkdir -p ~/.ssh to make a directory to hold our SSH configuration files. Next run chmod 700 ~/.ssh so that only the git user can read and write its contents. Then create a file to hold the authorised keys with nano ~/.ssh/authorized_keys and copy in any public keys you want to allow to login to this user. Finally restrict it to only the git user with chmod 600 ~/.ssh/authorized_keys

Setting up the git repo

Now that we have a user configured, we can make a git repo to hold our website. First make a folder for it with mkdir -p ~/website.git (you don’t have to call it website, but it’s good practice to end it with .git), then enter it with cd ~/website.git and initialise the repo with git --bare init. This is slightly different to how you might normally create a git repo. The --bare tells git not to create a working directory for it, and instead just make the files that would go in the .git folder directly in ~/website.git. We do this because we aren’t going to be doing any development on this repo, that’ll be done on a local clone, so we just need the history.

With this we should now be able to clone it locally with git clone git@your_server_domain_or_IP:website.git

Building and deploying automatically with git hooks

Git provides a convenient way to trigger scripts when certain events occur called hooks. In our example we can find them in ~/website.git/hooks/. By default there are a bunch of sample scripts in there that we can ignore. Instead we are going to create a ~/website.git/hooks/post-receive script (use whichever editor you like). I’m going to write the script in Python 3 which can be installed on your Debian/Ubuntu server with sudo apt-get install python3 python3-pip. If you want to deploy the website to a separate hosting over SFTP, you’ll also need to run sudo pip install pysftp.

#!/bin/python3

import sys, os, subprocess, shutil, pysftp

The first line here is the ‘shebang’ that tells linux that this script should be executed using the Python 3 interpreter. After that we have all the library imports that this script will use. If you don’t need to deploy over SFTP, you can remove , pysftp from the end of the line.

def check_refs():
    for line in sys.stdin:
        old_hash, new_hash, ref = line.strip().split(' ')
        if ref == "refs/heads/master":
            return (True, old_hash, new_hash)
    return (False, None, None)

Next we have a function that will check if the master branch has been updated as part of this push. If you want change which branch it is sensitive to, change "refs/heads/master" to "refs/heads/branch_name".

def make_temp_dirs():
    try:
        os.mkdir("/tmp/website_checkout")
    except FileExistsError:
        shutil.rmtree("/tmp/website_checkout")
        os.mkdir("/tmp/website_checkout")
    try:
        os.mkdir("/tmp/website_build")
    except FileExistsError:
        shutil.rmtree("/tmp/website_build")
        os.mkdir("/tmp/website_build")

This function will create temporary directories to store the checkout of the pushed branch and the results of the Jekyll build.

def git_checkout():
    subprocess.run(
        ["git", "--git-dir=.", "--work-tree=/tmp/website_checkout", "checkout", "master", "."]
    ).check_returncode()

This function uses subprocess to invoke git to create a checkout of the pushed branch in our temporary directory. If you want to make it work on a different branch, change "master" to "branch_name".

def jekyll_build():
    os.environ["GEM_HOME"] = "/home/git/gems"
    os.environ["PATH"] = "/home/git/gems/bin:" + os.environ["PATH"]
    os.environ["BUNDLE_GEMFILE"] = "/tmp/website_checkout/Gemfile"
    subprocess.run(["bundle", "install"]).check_returncode()
    subprocess.run(
        ["bundle", "exec", "jekyll", "build", "-s", "/tmp/website_checkout", "-d", "/tmp/website_build"]
    ).check_returncode()

This is the function that actually performs the build. It sets up environment variables, then makes sure all build dependencies are installed, and then executes the Jekyll build with the source and destination directories set to the two temporary directories.

def rmdir(sftp: pysftp.Connection, path: str):
    if not sftp.exists(path):
        return None
    for p in sftp.listdir(path):
        if sftp.isdir(f"{path}/{p}"):
            rmdir(sftp, f"{path}/{p}")
        else:
            sftp.remove(f"{path}/{p}")
    sftp.rmdir(path)

You’ll only need this function if you want to deploy over SFTP. The pysftp lacks the ability to recursively delete a directory, so we use this function to provide that missing functionality. This will be used to delete the old deployment before copying over the new one

def get_sftp_password():
    password_file = open("/home/git/sftp_pass", "r")
    password = password_file.read().strip()
    password_file.close()
    return password

Another SFTP only function. This one reads the password to use from a file.

def deploy(password: str):
    with pysftp.Connection("sftp_host_domain", username="sftp_user", password=password) as sftp:
        rmdir(sftp, "/home/sftp_user/www")
        sftp.mkdir("/home/sftp_user/www")
        sftp.put_r("/tmp/website_build", "/home/sftp_user/www")

This function deploys the website over SFTP. You’ll need to change "sftp_host_domain" to be the domain of the host you want to deploy to, and any occurrences of sftp_user to the appropriate user. You may also need to change www if you want it to end up in a different directory.

def cleanup():
    shutil.rmtree("/tmp/website_build")
    shutil.rmtree("/tmp/website_checkout")

This function is just a simple clean up function that deletes the two temporary directories once we’re done.

def main():
    should_update, old_hash, new_hash = check_refs()
    if not should_update:
        print("Build and Deploy: nothing to do")
        return None
    print(f"Updates master from {old_hash} to {new_hash}, rebuilding Jekyll")
    print("Making temporary dirs")
    make_temp_dirs()
    print("Checking out master")
    git_checkout()
    print("Building Jekyll")
    jekyll_build()
    print("Deploying via SFTP")
    deploy(get_sftp_password())
    print("Cleaing up temporary dirs")
    cleanup()
    print("Build and Deploy complete!")


if __name__ == "__main__":
    main()

Lastly we have the main function that calls our functions in sequence, with some simple logging in between, and the __name__ == "__main__" to run our main function when the script is executed.

All that’s left to do is make it executable with chmod +x ~/website.git/hooks/post-receive, make the SFTP password file at /home/git/sftp_pass and set it to only be read and writeable by the git user with chmod 600 ~/sftp_pass.

You can download the full Python script here