This a monthly development update for my mind-erasing stealth game Noirmancer. I usually give some brief status on how development is going and ramble about some aspects of game dev I've been thinking about.

STATUS UPDATE

These past few weeks I've been going through this most recent level and dressing it up a bit and filling out the loot. Trying to generally get it towards a presentable state so that I can make a demo in the not too distant future. Fortunately, there's not really been any surprises in that. And has largely gone how I expected.

The ideal process for making a game would be making the whole thing in placeholder art first, but I can't deny that it does a lot for my motivation to see a level get to a presentable state. It feels a lot more real. I also kinda feel that presentation affects your feelings about a gameplay experience a lot. It can be hard to really test some games without the aesthetic captured to get the right vibes. Visuals aren't just for eye-candy either, they can be important for gameplay.

VISUAL COMMUNICATION

So like many stealth games since the release of Thief: The Dark Project, my game has some floors that you will make loud footsteps on. It's nice for creating some variation of danger in a space, as long as you don't overuse it. Another danger when using it is making sure it's obvious to the player that the floor they're about to step on is a loud surface. My game doesn't make use of a lot of textures, I'm trying to keep a quite simplistic moody art style where I don't feel that fits. It also happens to save me a bunch of time which is useful as a solo developer. This means lots of floors are just the material that I use everywhere, and therefore, meaningfully distinguishing between a hard surface and a soft surface isn't something I get for free. So I've decided to try building up some visual design language that the player can learn to recognize to help them out.

If you see some kind of diagonal detailing on a floor surface, that surface will almost certainly react to your footsteps in some notable way. Diagonal tiles are the obvious one. I'm not entirely settled on it yet, but I'm also trying out using these little tufts in carpeted areas to try to suggest some slightly worn down carpet fibers:

The idea is to put these in anywhere where the surface changes to help the player identify what they're getting into.

DESIGN RAMBLE

Y'know, I imagine there's someone out there working an office job, staring at excel all day and hating spreadsheets, who loves video games and wishes they were making video games. Well, hypothetical person, I have some bad news: You may end up spreadsheeting anyway. Creative work is all about decision making, and you need some basis to make your decision otherwise you're just kinda thrashing around. And when that design involves numbers, that means accounting. And accounting means spread sheets.

I feel like the roll of spreadsheets in game design is a little unknown to most game players. So I'm gonna ramble about it.

So what's next

So, this level is finally broadly wrapping up. I'm sure I'll find some random stuff I want to change but it is a playable level. Next up I want to design out and build a short little tutorial, that also acts as the hook into this mission. In terms of working towards a first little public demo, here's what I'm thinking about:

• Make a short tutorial level
• Make some semi-okay start screen (the current one is a black screen and two buttons, lmao)
• I currently only use one set of NPC graphics, I'd really like to have another 2 before releasing the demo but it might be better to just swallow my pride
• I feel like I'd like to replace/expand the vocal sounds I currently use for the NPCs, but I may just end up skipping that.
• Make an ability menu that works with only the few abilities that'll be featured in the demo.
• Any random changes to the level that I think will really make a difference, particularly based on some private playtesting.

Follow along

Wishlist the game on Steam. You can follow this project via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email, or Mastodon with any comments or questions.

LAST TIME ON NOIRMANCER DEV BLOG:

I'm still on pace to hopefully have a demo out in late July or early August.

Uhhhh, yeah that ain't happening.

This a monthly development update for my mind-erasing stealth game Noirmancer. I've decided to not be precious and show you some screenshots and clips of areas that are very unfinished, so please take the opportunity to smugly judge as much as possible.

STATUS UPDATE

So I spent the time since the last post working on this new level. All the rooms in the level are blocked out and have their distinct bits of gameplay implemented although I still need to fill most of them with actual intel (loot), that's not the hard part though. All the level's objectives are rigged up. While most of the level is still greybox and underdecorated, I have started working a bit on the aesthetics of the exterior:

I got to implement a bunch of fun little ideas for the level, including the unique objects mentioned in the last blogpost. One idea that I had thought of a while ago and wanted to incorporate into the levels at some point was the ability to escape a space via turning to mist and getting sucked up an extraction fan:

As with every month, the game's repertoire of pre-built gameplay components has increased. So I end another month with a more fleshed out game than I started.

But, I'm behind where I wanted to be. Why? When I left off last time I was still ideating about gameplay ideas for different parts of the level and that took me longer than I hoped, so that was one part.

Last time I talked about making some unique objects with unique effects related to the level's story as a way to give this level some unique toys when it otherwise lacks many of the game's supernatural powers. But the objects I chose affect NPC behaviour and so it required a fair bit of programming to get them in, which caused a bit of a slow down.

And third was just I was fairly unproductive. This was surprising to me because I had been really productive in the 2 months before launching the game's steam page and my expectations were for that to continue. But no, I really dragged my heels on a lot of this. Perhaps part of it was a winter slump (I'm in New Zealand) but I suspect a big part of it is my personal tendency to hesitate in the face of uncertainty. This was really annoying because again, it wasn't a problem I encountered in the 2 months before the Steam page launch, and I wasn't expecting it to be a problem during this phase of development either. This is something I need to be stricter on myself about; just bloody do it. It's pretty rare that I actually find a problem that turns into a real quagmire. Even my design gaps tend to be resolved quickly when I commit to sitting down and not getting up until I have a solution.

This past two weeks has been fairly productive again in part because I went back to my old faithful technique of self-manipulation: the pomodoro technique. It just really clicks with my brain; "Clock says I gotta stay working!". I started using this technique a number of years ago at a job and I've always found it effective. But for whatever reason I've been bad about employing it in my own endeavours? That seems... stupid? I guess in summary, I am a stupid baby.

DESIGN RAMBLE

For this post's design ramble I talk a bit about ability design. If you just wanted to know how the game was going, feel free to skip it. Or, click below if you too enjoy overthinking things:

So what's next

So the next week or two will be making the level not a whole bunch of grey, filling it with decoration and actually putting down all the intel.

In terms of moving towards a first public demo, there is sadly something else I must do. I decided I didn't want to burden this level with introducing the core gameplay. That means I need to make a short linear tutorial level that'll introduce the basic ideas and the hook for this level I'm working on right now. So I think at the moment I'm hoping for early September for that demo. Which sucks, but that is the reality of it. I'm glad I started working towards that much earlier than I really needed to. Hopefully I'll feel more certain about when that is coming in the mid-August update.

Apologies this update is at the last minute for July. I decided I at least wanted to get all the unique gameplay stuff and mission objectives in for this level before bothering to make a post in which I mostly just said I wasn't done.

Follow along

Wishlist the game on Steam. You can follow this project via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email, or Mastodon with any comments or questions.

Time for an obligatory monthly update for my mind-erasing stealth game Noirmancer. In this blog post I ramble a bit about what I've been working on in the past month, and some thoughts about level shape. If you don't feel like reading, a TL;DR is that horse-shoe = good and that I'm still on pace to hopefully have a demo out in late July or early August.

Where do you keep your lucky horseshoe?

Dopamine Economy In Crisis

Sadly, I have no funny moving pictures to show you for this month. Apologies for the shortfall in your projected dopamine activation. This isn't going to be the most gripping update but hopefully July's one will see your brain engagement juices skyrocket, so look out for that.

Let's get the boring stuff out of the way: I reworked some guard AI search point selection to remove some manual work from me and also improve it generally. I did some performance work that I had procrastinated. Specifically tooling stuff related to merging a bunch of level geometry brushes to reduce the number of draw calls. I just wanted to check that it would work out the way I hoped so that I can go forward with making levels with the knowledge that that should be all good. I also upgraded the version of Godot I was using. This conveniently solved a crash I occasionally was encountering right around the time of the last blog post.

Story Time

After doing that, I spent quite a while working on writing stuff, mostly being creatively blocked on writing stuff specifically. I had procrastinated a lot of writing decisions in favour of focusing on general game-play up to this point. I had many ideas but not that much concrete, but I've settled on a goal for the story now. It may change later, but I'm optimistic and it gives me something to go forward with. This was important as part of starting the design for upcoming public demo level; I needed to know what story the demo was going to tell.

Finding that story was also key to solving a game-play problem: I want the player to have more abilities during the course of the game rather than just dumping them all on the player immediately. Which means less game-play ideas to design the level with, and there's a concern about there not being enough variety in the level.

My solution to this is to have some unique objects with some unique effects in this level, related to its story. It both helps make the level more distinct generally and can help with the shortfall in player options created by the lower number of abilities.

And no, I won't be spoiling the story for now. Sorry :)

Level Shape

With my level's story and game-play identity decided, next I needed to define the geometric identity. This is often called a parti. I don't know if the parti of a level is really noticeable to the player, but in the worst case it's at least a helpful tool for the designer; it gives you something to fill in. And while I'm not sure players can really identify the parti, I suspect they'd realize if your levels all have the same shape. So it's good to make sure you use a different one each time.

At first I kinda wanted to do a ring-shaped building and you emerged in the court yard in the middle at the start. This design means I can spread some ingress points around the player's starting location and give them very different angles of attack for them to discover and choose from. But, since this level is a key early level, and meant for this public demo, I kinda want to give the player a cityscape vista, so I decided to break the ring shape, producing something more like a horseshoe.

Now, horseshoe shapes are NOT uncommon in level design. Here's E1M1 of Doom.

Thanks to Doomer GCP for the isometric map

Here's E1M1 of Quake. Taken from the models resource

I noticed a familiar shape in the first level of Thief 2 last time I played it.

John Romero has talked a bit about the Horseshoe before. My usage of the horse-shoe is a little different though. I think the reason why it works well often for first levels is that it's basically as simple as possible without being boring; it's a line, but a curved one. So you don't feel like you're just walking down a line. It's easy to position some little tangents coming off it, easy to allow you to see the start of the level from the end which is always a strong feature in level design. But I'm using it for the previous mentioned goal of surrounding the player different ways to get inside.

I'm enjoying getting to play around with breaking symmetry. Symmetry is very normal in many of the large buildings I look at for architectural reference but it can also be very boring. The obvious tricks are about having symmetry but avoid homogeneity; use a broadly symmetric shape but fill that shape with extremely different components, or subdivide them in different ways.

The Next Month

So, at this point I have the core ideas for the demo and have the rough shape of the level blocked out in the editor. I'm now going through and breaking it up into smaller rooms and thinking of the game-play ideas to fill them with. Sadly, the ideation process has gone slower than my last time making a level, I think a large reason is that I simply feel a lot more pressure for this one to be good. The July target still seems doable though.

Follow along

Wishlist the game on Steam. You can follow this project via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email, or Mastodon with any comments or questions.

Long time no see. My mind-erasing noir stealth game previously codenamed "MaskedDetective" lives. And I've been working on pretty consistently for the past year. There's a Steam page up now!

WISHLIST IT. CONSUUUUUME

State of the game

The game has basically passed the vertical-slice. The core gameplay loop is in: guards wander around doing their stuff, you get spotted, run away, use your mind erasing powers to reset them. You collect intel, keys, read notes. Discover new objectives mid-level. You can use nine different powers to break-in, sneak around, and lose guards. There's a bunch of interactable props and other features to make levels from. I have made two complete levels and playtested both of them. Many things have been refined and reworked based on that progress. It's not as nice-looking as I'd like, but the art is generally representative of the intended look. It needs a lot of fleshing out, but it's a playable, real game.

Big shout out to artist Giovanni Fim, who I commissioned to make this promo art

What's Next? Public Playtest

I've spent the past two weeks since getting the steam page up doing some tasks I had neglected in favour of doing that. A big one was reworking some of the guard AI; it's a tough thing to get right and I imagine it'll probably get continously revised throughout the development of the game. But I'm ready to target the next milestone.

I want to make a small playtest demo and get it out there. It'll allow people to see what the game is really like, it'll hopefully allow me to get feedback, and it'll be useful in terms of promotion: having something playable can be very helpful for coverage and access to events.

I'm thinking a first playtest may need to be two levels, a short one that introduces core gameplay ideas and a second one that is a bit meatier. The last level I made took me less than 3 weeks, but I probably want to be a little more ambitious for this one; flesh out the game a bit more, take some time to refine it, etc. So I'm hoping to have something in July but obviously that'll depend a lot on how the creative process goes, how ambitious I get, how things go in an initial private playtest, etc.

I intend this to probably be the first of at least two demos. This first one will probably be a bit rough; it's about getting feedback and building interest. And there'll probably be another one for a Steam Next Fest closer to release. It probably wouldn't fit into the timeline but it would be great to do one in between those two; that would hopefully be a refinement based on feedback from the first one, but yeah it probably can't be done time-wise. I quite like the idea of developing a game with regular playtest demos; it's kinda like doing Early Access without the perils surrounding it.

Where the hell have I been

I have not touched this blog in ages. There's a lot of reasons for that. I spent many months with my focus elsewhere, and obviously a lot of time working on the game. But a big part of that time was me working through my pretty bad relationship with the creative process. I had some really bad priortization. I didn't push hard enough. And probably most importantly, I allowed myself to get paralyzed by uncertainty far too often.

Definitely heaps of good work happened in that time, but it happened at a far slower pace than ever should be ideal. Recently I feel I've gotten better at all the mentioned areas: priorization, producitivty, and working through uncertainty. Creativity can and should be very playful but sometimes it's work and I feel like I've finally gotten good at the work part.

Additionally, I really wanted to work on the game and work through those problems rather than just blog and social media post about it. And I think that was a good choice. I'm now ready to talk about the game, because it now feels pretty real and inevitable.

I'd like to blog more. I want to commit to a monthly update. So I guess you'll be hearing from me again in June.

Follow along

Wishlist the game on Steam. You can follow this project via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email, or Mastodon with any comments or questions.

Hi, I'm working on a stealth game were you play a comicbook detective who uses superpowers to sneak, spy, steal, and escape. Last time I set myself the goal to work on a first test level and blog about it in December. This was a terrible choice given uhhh, its december and I was moving. But now that things are settling down a bit more I can get back to it.

In this blog post I want to start sharing my process of attempting to create this test level, and talk about the power of the Design process.

(© Warner Bros. Animation)

Form Follows Function

Creative tasks like Level Design are plagued with one famous problem; the blank-canvas. The simple lack of knowing what to make. How do you know what to do from infinite options?

In practice the options aren't quite as infinite as they seem. You probably want to make something that at least you consider good, and ideally others do too. Additionally, you're trying to make a good level for your game. Doom's E1M1 is a great level, but it likely wouldn't be a good level for your game. You're also probably not trying to just make any good level for your game but perhaps a good first level, or a good level following a certain twist in the story, or after the introduction of a particular mechanic, etc. These elements constrain the possibilties. You're not just being creative; you're trying to fulfill a need.

This is the essence of Design: "Form Follows Function". Identify the function of what you're creating, and create to fit that need. Now you've gone from what felt like aimlesly considering infinite possibilities to working towards a defined goal. Design isn't some magic of summoning an idea from the void; it's a process, like Science is a process.

Promises and Values

A good place to start thinking about design for a game is to specify the Promises and Values of your game. Promises are the hooks, the pitch, the thing that gets players to play. They are an incredibly important part of the relationship with your audience; without making promises the audience simply doesn't know if they care about what you've made. They want to know if it's something they're interested in. What are my promises? Well I've already been repeating them in this blog series.

• Comic-book feel and aesthetic
• Detective
• Stealth
• Powers

They're simple ideas that can be easily communicated, and someone can instantly start to get a sense of whether or not they're interested. They're very important context for designing your game because if you make some promises to your audience and they don't feel like you fulfilled them, then they're going to find your game very unsatisfying. In fact the same piece of content with a different promise can receive radically different appraisals from the same people.

Values are different; they are more nuanced. They're not really meant to be in marketing material, instead they are opinions about how your game should be. They're a philophy. A subjective taste or a theory on how to make something good. Sometimes they're near universally accepted theories such as "the main character of a story should have goals and things they care about". They constrain and guide you. Here are some values for this game:

• Feel sneaky even after being discovered
  • (I cover this as a problem I have with stealth games in the first blog post)
• Every mission is a story.
  • They should be surprising, develop in complexity, unfold.
• The levels should say something about the fictional world.
• Exploration is key
  • You should generally be at least a little non-linear.
• Navigate without UI markers or a map screen.
  • I really dislike pausing to check the map screen in Thief, it ruins the flow of the game. As much as possible I want the player to be able to orient themselves using cues from the environment. And probably just remove the map screen entirely.
• Every Level should have a unique identity, and have its own hook and interest

By identifying my Promises and Values, I now have a lot of context to feed into this test level.

Level Function

So, what's the function of this level I'm making? There's a number of things I want to achieve.

• Show off all the promises of the game
  • I need to test that I can deliver on the promises. Ideally this is also a hypothetical "first level" for the final game (unlikely to end up being so in reality) and a good idea for a first level in a game is to do a little bit of all of your promises. Allow people to understand what the game is really about.
• Teach the basic gameplay, including the memory erase feature
  • The memory erase feature is meant to be a big value of the game and a core part of the gameplay loop, the player absolutely needs to learn it.
• Introduce a world and a setting
• Be surprising. Wow the player a little.
• Interest the player in seeing more of the game.

By fleshing out these requirements I can start turning my task from "Make a level" into the more specific task of "Make a level with these particular properties." I'll try making a list.

• Things to communicate
  • The player is a vigilante detective
  • The world does not look kindly on vigilante detectives
  • The big corps of the world are in-fighting. Having successfully co-opted or suppressed regulators and unions their only opposition is now each other.
• Key moments
  • Feel very sneaky
  • Use some cool powers
  • Discover something and feel like a detective.
  • A "twist". A moment where everything gets a bit more complex. Things can't just go according to expectations. Ideally this moment changes the pacing of the level, which also often means the gameplay.
• Things the player must learn
  • The basic controls
  • How to sneak
  • How to interact with objects
  • How to erase memories
• Some key details
  • The player should always have some kind of goal, give them a mission statement right away. But as mentioned above, there needs to be a twist, there should be more to it than the original goal.
  • A feeling of mystery
  • Feel like a comic-book, or specifically Batman the Animated Series.
  • Some non-linearity, to help you feel like you're exploring.
  • Has a unique identity. Something about it that no level after it would be allowed to do.

With this list I can begin to brainstorm specific ideas of how to do each of these things, and build things into a layout.

When you look at this list you can begin to identify some things that seemingly must be true. I have some things the player must learn, which means they probably need to be the first things in the level and there needs to at least be a bottle neck to prevent you from getting into the meat of the level without learning those. This "comic book" or "Batman the Animated Series" specific feel is interesting. I really associate BTAS with its cityscapes, which really suggests I should do a cityscape level. Which could be the unique identity this level gets to have. I probably don't want to do too many cityscape levels.

For it to be a cityscape there needs to be multiple buildings. So I probably need to avoid putting the player into one specific building for too much of the level. And should probably make use of a lot of big windows, balconies, walkways, and rooftops. If you are going between multiple buildings I probably want there to be some cohesion between these buildings world-building wise. Ideally I want things in some building to relate to things in another building. And I probably want the player to feel like all the buildings collectively make the level a single shared space, rather than just the levels being a series of individual buildings you break into one after another.

In terms of the relationship between buildings, I do have an option here I can pull from else where in the list: "The big corps of the world are in-fighting" perhaps this is some CBD area of a big city and there's several competing corps all within a block or two of each other, and your infiltration into these buildings leads to you realizing the extent of the rilvary between these companies. Perhaps even between their owners as people. I can put penthouses in as a neat difference in space and to pull in those more personal elements of their owners or employees. And while I do want the buildings to have a story relationship, I don't want them all to look the same; each company could have its own architecture aesthetic or interior design aesthetic.

For the "twist" or big pace change, I can again look to that rilvary. What if one corp secretly hired a group to kidnap the head of another? They'll make it look like environmentalists or "Extremist Unionists" or whatever. Notably, if this happens while the player is in the building, then the security situation in that building instantly changes. Everyone is on high alert, which makes things very hard for you. But also you have a big corporate secret to try and find proof of.

There is an upside to moving this beat forward a bit into the level: perhaps the victim knows it's bullshit, and says something like "Who hired you? Was it <competitor name>? Maybe <another name>?" and this sets up the player to break into the near by competitors and identify which one has organized the kidnapping. This gives the player goals and can create some obvious non-linearity potential. However I probably want the increased security to be something later in the level. Perhaps the player has to circle back and when they get back, the kidnapping has been discovered by the personal security of the victim and a conflict is escalating. That sounds like it could work.

The above is a great example of the Design process in action. I identified the function of my level, and from only a few of those specific goals I have derived a setting or "stage" I guess you might call it, some basic plot, and a loose structure for the level. I went from a blank canvas to a sketch. Still a lot of work to do, but a big hurdle is overcome.

If you'd like to hear more about this sort of process from someone with real professional Level Design experience, you should check out videos from Steve Lee, who I've found extremely helpful.

Dev planning!

So what's next? The near-term goals are still:

• Get a few powers in
• Make a test level

This process has helped a bit with the first one: if I want to enable some rooftops gameplay, I may need abilities that specifically help with that. The second one still needs more design work, so the next post will simply be further refinement on this idea, and some early work thinking about the level layout.

(© Warner Bros. Animation)

Follow along

You can follow this project via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email, or Mastodon with any comments or questions.

Hi, I'm working on a stealth game were you play a comicbook detective who uses superpowers to sneak, spy, steal, and escape. It's been a productive time since my first blog post about this project, and I want to talk a bit about the AI in the game, particularly thoughts on searching behaviour both from the perspective of design and programming.

What have I done since last time?

A short video of the this very early Alpha, extremely subject to changes.

Since last time I finished some work on the AI around things like searching and their simple punch attacks. I also worked on a very early version of the memory-erase ability.

Game Design is often about push and pull; finding that sweet spot where the player feels they can influence their destiny, while still feeling contention from the game. For my memory-erasing gameplay I need the player to experience a simple loop:

• Sneaking
• Being discovered
• Evading
• Hunting

Sneaking and being discovered should be the easy parts, it's balancing the evasion and hunting steps that matters. First, I need the memory-erase ability to actually require "hunting". I decided to make it a short-distance ability that works only on enemies facing away from you. This means you can't just run up to an enemy and get them; you need to carefully stalk them.

Additionally, it means you can't immediately use it after discovery; you have to lose them first. That's the evasion step.

To help get the balance on this right, the NPCs run very fast if they can see you, go at a medium speed for a bit after losing you, but then slow down again a short while later as they're combing around for you. This way you're not having to chase an enemy who is always sprinting. The player is faster than the NPCs, but has a stamina pool that becomes visible after 20% depletion; this gives you a window of time to lose the NPCs.

The memory-erase ability causes the NPCs to become stunned for a period of time. This time period effectively acts as a soft-window in which you likely need to get all currently alert NPCs, otherwise they will just re-alert each other when they leave the stun. Getting the balance on that correct may prove challenging, but that'll be a future task. I suspect I may end up with a secondary memory-erase ability that is resource limited but a bit more flexible, to help with some of the stickier situations with multiple NPCs. The search behaviour tries to spread the NPCs out, which is both believable and gives the player a helping hand.

I expect this area of the design to need a lot of iteration as I develop the game, find new problems, try to make it more interesting, or try to find ways to keep it fresh. But for now, it works, and it's already a little challenging. So I'm hopeful that I'll be able to get something interesting as I refine it and flesh it out more.

Searching

Searching presents a number of interesting challenges to designers and programmers. It influences how players feel about the NPCs because it's very easy to write search behaviour that feels very stupid; in Thief for example the guards quite often search around in ways that have them mostly staring at walls from a point-blank distance. But on the other hand, if you make it too smart you actually create a problem.

About a decade ago I hit on this exact issue with a little Guard AI test I programmed for a top-down game. I wanted guards that felt like they were actually intelligent, and therefore made escaping from them feel special.

Rather than picking random near points to wander to a better approach seemed pretty obvious: when the AI loses sight of the player, track the space they could possibly be. I did this by dividing the world into a grid, and the AI knows the player had to start from that grid position. From there, you have the AI expand the possibility space in the grid at roughly the running speed of the player. You remove spots they can see from the possibility space, and obviously make the space only expand into grid positions that can actually be walked through (so not into walls).

This way, if the AI sees the player run down a dead end corridor, and has sight of the only exit/entrance, they'll know the player can only be in the corridor. From there, searching is as simple as picking a point in the possibility space (generally the oldest) and sending the NPC to go look at it. I recently learned that this approach appears to be quite similar to the one used by Dishonored 2.

Too Smart For Its Own Good

This works extremely well, and isn't too crazy for a computer to compute. But I had a problem: now the guards were too good at finding the player. The game requires the chance that you can get away, and AI that perfectly determines where the player can and can't be often lowers that chance too much! Giving the player powerful means of evasion can help counter this, but I found it difficult to do so in that old prototype without giving the player too much power. It seems particularly hard to balance evasion.

This is part of my reason for picking the memory-erasing idea. For my game the push-and-pull is more around the pressure to erase memories before time runs out, rather than the pressure to not be found. This means I can make the player pretty good at evasion, as they have to actively put themselves back into danger in order to erase memories.

So, I used a similar search approach for this prototype. However, rather than a grid, I used the points of the navigation mesh, and rather than slowly expanding the area, I simply give every near nav-point a score based on its proximity to the player's starting point. When the NPCs see a point they mark it as low-score, and the scores grow back up as long as they aren't being looked at. A key difference is this means the search area is never actually fully erased, and if they check every where they simply will start to re-check everything.

As with everything, there's a likelyhood of me tweaking this more later, but for now I'm quite happy with the approach.

Dev planning!

So what's next? The near-term goals at this stage are:

• Get a few powers in
• Make a test level

Obviously that second one is easier said than done. I'll also probably do other small tweaks and additions as I feel like it. In the next three weeks I'll make another blog post about my thoughts on designing this test level, and how it's going. I expect to be a bit of a mid-way progress report.

Follow along

You can follow this project via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email, or Mastodon with any comments or questions.

I've been tinkering on a project I currently codename "MaskedDetective", and I've decided to start journaling its development on here, in hopes that it'll help me organize my thoughts and maybe provide some interest to others. This post is about the project, its influences, goals, and how I hope to address what I consider a core problem with stealth games.

So what's the idea?

"A stealth game where you play a comicbook detective who uses powers to sneak through shadows, climb skyscrapers, turn into mist, and erase the memories of his pursuers, as he investigates dangerous powerful people."

It can take a while to find a good hook for your game and as a game evolves you often need to evolve the hook, but this is probably a decent starting point. I think it does a decent job of communicating the general tone, and gives a vibe of what the gameplay may be like. And in this case, hopefully gives readers of this blogpost an idea of what to expect.

Influences: what to copy, what to change.

The biggest influence on this project is the classic stealth game Thief, and probably more specifically its sequel, Thief 2. The bits of interest to me are its system-and-tool driven gameplay, its focus on stealth over combat, and how level design really sits at its heart.

Those familiar with the term "immersive sim" probably know what I mean by "system-and-tool driven gameplay", but for those who don't; what I mean is that Thief gives the player a bunch of different gadgets and puts them in a big map to explore and let's them go nuts. Levels aren't really about the player figuring out the specific way they're supposed to overcome them, but rather letting the player find their own way to infiltrate a space. In MaskedDetective the main character possesses and acquires a number of supernatural abilities that you can then use to find your way around the games obstacles, and I'm hoping to capture some of this same feeling from Thief.

A problem with stealth games, and how I hope to fix it.

I like Thief's focus on stealth over combat. However, there are some problems with how Thief does this. When you are discovered you either need to evade your captors or subdue them. And neither of these really match the reason I play Thief; I want to feel sneaky.

It's no secret that many players aggressively use quicksaves in stealth games to avoid ever being detected. And while this is sometimes perfectionism, I suspect a major motivator is that being discovered in stealth games is boring. I'm not playing these games to fight people so I don't want to do that. Waiting for guards to give up searching is tedious and takes the player out of the driver's seat; you no longer solve your problems actively, instead you wait for them to go away. I don't think player's use quicksaves here because they can't handle failure, I think they do it to get back to having fun.

Fortunately the idea of using supernatural powers affords me a lot of flexibility, so here's a unique gameplay twist; you have 60 seconds to use your powers to erase the short-term memory of NPCs, and therefore make them forget that they saw you. But if you fail to do so in time, then you have failed the mission.

So now rather than having a gameplay loop around Sneaking->Discovery->Waiting, it's Sneaking->Discovery->Erasing. And while it'll take some effort to make sure the memory-erasing is fun, it should be sigifnicantly more enjoyable than waiting in closets for guards' goldfish memory limit to kick in.

Level Design is king

Something that I think doesn't get addressed enough with Thief is how much of the appeal is about levels. I think the audience understands this on some level; it is reflected in the fact that the games still have a community making levels for them today. But I don't think I've seen anyone else say it so concretely; Thief's stealth gameplay is just a really good match for exploratory levels. You really find yourself focusing on the space. Looking for hidden things, ways to access areas, spots to hide. Additionally, the somewhat voyeuristic nature of secretly reading notes and letters lines up with the wider sneaky-fantasy that this kind of game sells.

Level Design is an artform, one that I have little-to-no experience in, but I hope to do my best in creating levels that are fun adventures, and really act as the heart of the game.

The amazing backgrounds of Batman The Animated Series

The goal that I expect to struggle with the most— and to fall very short of— is to try and create a 3d game that evokes the aesthetic of Batman The Animate Series' amazing backgrounds.

I'm very underequiped to pursue this task, but I have some thoughts that I plan to share in later blogposts.

So, where is the project currently?

Very early days. The character controller is in, visibility levels are calculated, some early work on the AI, readable notes, lockable/unlockable doors, a WIP power or two. This is very must a pile of tech and feature tests at this stage and not a game. I hope to use these devblogs to organize this development a bit, speaking of which:

Dev planning!

The current near term goals are:

• Have basic NPCs that can detect the player, chase them, search for them, subdue them.
• Implement memory-erasing gameplay and lose-condition
• Get a few powers in
• Make a test level

Technically speaking, once you have a game where you can hide and be found, you have a stealth game. But I need to make a tiny test— dare I say, "a vertical slice"— of level design, powers-driven sneaking, and that Sneaking->Discovery->Erasing loop. Game design is a process of hypothesis->test->iterate; I need to check that I can make that memory-erasing gameplay fun at all.

Working on the aesthetics is not really a goal in the short term, so please expect screenshots to look AWFUL.

Let's set a goal, in the next two week I'll post about the NPCs. I already have them pathing, patrolling, detecting the player, chasing the player. And by then I hope to have them attacking the player, and a reasonable first-pass at their search behaviour. Both of which I'm already working on.

Follow along

You can follow along with this effort via RSS (Don't know RSS?), or by following me on Mastodon. And of course feel free to contact me via email with any comments or questions.

(In this blog series I attempt to develop a wiki server that tries to be respectful of computer resources, as part of a self-assigned Modest Programming Challenge, the series starts here)

A while ago I set out to make a small wiki ideal for niche communities. Easy to administer, reliable via low number of parts, respectful of computer resources both on the server side and on the client side, with little code to make it easy to pick up and maintain on one's own. It's been a while since my last update but not because I stopped the project. Actually, the wiki is up live and has been for months. So how have things turned out?

I never said I was a webdesigner btw. But you can probably tell that from my blog.

Mission success

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          15            612            467           1821
HTML                            24             14              0            792
CSS                              3             83              1            395
-------------------------------------------------------------------------------
SUM:                            42            709            468           3008
-------------------------------------------------------------------------------

Mosiki has shaped up reasonably well. It has users, sign-up via an invite system, password reseting, page creation, editing, conflict management, permissions, renaming pages with URL redirects, image uploading with thumbnail generation, markdown syntax with some additions, some simple mobile responsive css, page locks, and page keys.

At idle the server uses about 90 MB of RAM, including mosiki, OS (Alpine Linux), and Nginx proxy. A reasonable example of a fairly simple page tracks in at 22KB of HTML and CSS, with obviously any attached images dwarfing that. There's no JS anywhere. It comes out at an absolutely tiny 3k lines. I feel like I can very confidentally call this some Modest™ software.

Before I finish up though, here's some more technical junk.

Yet more dependencies

Last time I went over some early choices: Flask, Gevent, Markdown via markdown-it-py, SQLite, Argon2. And all of those have ended up in. I did end up adding in a few more though.

mdit-py-plugins

This module comes from the same org as markdown-it-py and allowed me to quickly add anchor-links (as in, linking to sections of a page) and footnotes. This is perhaps one of the more dubiously low-value dependencies of the entire project, but it didn't seem fundamentally unreasonable.

I also added spoiler sections and other minor markdown tweaks myself. Additionally I found a bug in the module and submitted a fix so this silly little project has officially contributed to the ecosystem.

libvips (via the pyvips binding)

I knew I wanted image uploading in the wiki, and it seemed reasonable to want thumbnail generation to save a little on bandwidth. This is the sort of thing that when designing your software should ring bells as a potential memory hog point; if you're not careful it's easy to end up in a situation where your image processing solution loads the entire image into RAM and then processes it. Several megabyte images are fairly common, and so this could easily consume a lot of RAM with a large userbase. In practice I won't have many users so the only concern would be a denial-of-service attack, which is also very unlikely. But, this is easy enough to work around that I might as well anyway; it's good to take pride in programming resilient solutions.

Image processing is a complex enough problem that you probably should use a dependency and you should probably just pick one that removes the memory hog vector. I went with libvips. Libvips is much faster and less memory intensive than other solutions. The speed is largely due to use of SIMD, and the low memory usage is because it works on streams. This allows me to feel some confidence that Mosiki should be reasonably resilient to high file-uploading load. Honestly much higher than I ever expect it to experience.

The one thing to be careful about here is that libvips is written in an unsafe language, runs on client-controlled input and this is a server. In this situation you should always at least check that the library is fuzz-checked which will help identify security issues. Google, as much as I take issue with them as a company, does a nice public service with the oss-fuzz project and they fuzz test libvips. I checked its history of CVEss and it seems reasonable. Obviously not perfect but it seems reasonably infrequent and lots of what's there is specific limited danger issues. Hopefully distribution package maintainers stay quick on the draw there. Contrast ImageMagick which seems like should never be used in a server.

Diff-match-patch

Another obvious area of potential performance concern was text-diffing on conflict. Python provides a semi useful diff module called difflib, but it's quite slow. I considered writing my own stuff on top of it to recursively break the sequences up in order to prevent any individual diff-step getting to complex and slow, but honestly, this is the sort of thing you want to get right. And Google's diff-match-patch module seems very reasonable. It's a single file, although reasonably complex at 2k lines. It's pure python but much faster. And even supports a time limit after which it gives the diff to the current-best granularity.

An example of a diff on Mosiki's simple conflict page.

The module hasn't received any updates in three years. I don't consider that a bad sign. It's something with a closed scope. It simply doesn't need to do anymore, and it's something that can be verified reasonably in-depth via generative testing. As such, I have simply included the module in a "thirdparty" folder. I don't see any reason not to as I don't expect future fixes or security problems, and it's a single file. Package managers like pip are hoenstly themselves a quite dangerous point of security failures and I feel when it seems reasonable to just ship the dependency you should consider it, because package hijacking does happen. Relying on distribution package maintainers though seems to have a good track-record and so getting libvips that way for example likely increases security over shipping a fixed dependency.

And that's it. The total dependencies for this at this stage seem to be:

• Sqlite (provided by python but installing it separately for the command line programs is still helpful)
• Flask
• Markdown-it-py
• Gevent
• Argon2-cffi
• Nginx
• Mdit-py-plugins
• Pyvips/Libvips
• Diff-match-patch
• zope-interface (a dependency by something else here but I forget which one)

I feel these generally all pull their weight with maybe the exception of mdit-py-plugins.

No email

To simplify the needs of the server, no workflow in the wiki uses email. Instead users sign up via invite codes given by admins. If a user can't log in they can ask an admin to generate them a password reset link. Obviously this adds a bit of overhead but for a niche community I suspect it's reasonable. Using emails adds a whole other layer of stuff that can go wrong, and this project was specifically about making the minimal thing for my use case.

Iframes, really useful for JS-less pages?

Developing this without JS was interesting in some areas. For example, with image uploads I don't really want the entire page refreshing just because you uploaded something. The solution I went with was just that the uploads are handled in an iframe, and the iframe refreshes. And honestly it just works very well? I'm not a front-end guru though so maybe this is weird, feel free to send me scathing emails to educate me.

Accretion-only db

As a general rule, the wiki doesn't use much of UPDATE or DELETE. Instead new records are inserted and differentiated via an "updated_at" column. Current is selected via "SELECT * FROM pages ORDER BY updated_at DESC LIMIT 1", which indexes should make reasonable. This is a slightly unusual choice but I find it a safer design in general. Additionally, you usually want to keep records of what things were anyway, and this is just a very simple way to achieve that. The most notable exception is passwords, of which I only keep the most recent.

Hard limits in software are good, actually.

While programming I identified a few places where large amounts of requests could blow the server up, such as conflict resolution, image upload, password hashing (because of argon2). While this will likely never be a problem I actually have, I decided to go ahead and address them anyway. A big part of this experiment is being respectful of computer resources, and I think it takes more than just measurement; you need to proactively consider what could go wrong, and how to deal with it.

Image uploading was largely handled by libvips and wrangling file uploading to use streams rather than keeping the entire file in memory. But for password hashing and conflict resolution I decided to go ahead and use a lock to simply limit it to one request doing such work at anytime.

This may seem like an odd thing to do and yes, one at a time is definitely going to be unnecessarily limited for many real-world use-cases. Fortunately, I don't really have those use cases, but I would also argue that limiting the number of memory-intense requests is a good thing. It's tempting to write software without hard limits, but I think this is an illusion. You do have hard limits. At some point, memory runs out. If you want to feel reasonably confident that your memory usage will be low you need to pick a target, and the only way to reliably hit your target is via limits.

Things are boring when they just work.

It took me a while to get around to writing this update for perhaps the best possible reason: honestly everything went quite boringly well. The server went up (warning: nerd shit, but it's not much to see at the moment) and has stayed up for months. It requires no effort on my part, other than the occasional OS update for security patches. I should count this a total success, but man, it makes for boring blogging!

Despite the fact that I tried to write it small in order to be understandable, I'm not entirely sure that I'll ever share Mosiki; The point of this blog series was not "hey, you should use my wiki server". It was an experiment in a way of thinking, and an attempt to promote that way of thinking. I successfully made something that is useful for me and I basically don't have to worry about it, and that is honestly kind of unusual with software, but I think it shouldn't be.

I think it'd be possible to have more things that Just Worked™ if we valued different things. Conscious scope management, careful technology choices, respect for computer resources. Not everything can be as simple as my toy wiki implementation, but many could be a lot simpler than they are. And a lot more mindful of where things could go wrong.

This may well be it for this blogging series. Thank you, dear reader, for following along. I hope you found it interesting. And I hope to have more things to blog about soon that are a bit more interesting than a toy wiki.

(In this blog series I attempt to develop a wiki server that tries to be respectful of computer resources, as part of a self-assigned Modest Programming Challenge, the series starts here)

Alright so I want to develop a wiki— which for now I shall call Mosiki— with speed and lightness. A sensible first question is what am I gonna be writing this in? The tried-and-true high performance language C++? The classic and original C? Or perhaps rewrite all of modern software itself in Rust?

But no, I'm probably just gonna use Python.

Let's just drop our standards way low now

Now, there's a chance someone saw the first post and went "YES! You're right! All these lumberjack programmers need to stop using JS and transition to something actually capable of making good utilization of resources!" and that sort of person would now promptly call me out for being a poser right now.

I do want to clarify, if you want to truly approach the height of what is really possible with a computer, then you're almost certainly going to want to use a statically compiled language. One where you can reason about how your program is actually going to work in memory, care about cache locality, and maybe break out some SIMD instructions. Certainly if your language has some kind of runtime, uses garbage collection, and a JIT, it's literally doing more than is necessary to actually just run the program, of course it's not going to be as resource efficient as something that isn't doing that stuff.

But no. I'm not gonna use one of those languages. I'm not even gonna use one of the fast JIT languages. I'm gonna try Python. The thing with the damn Global Interpreter Lock that has prevented good multiprocessing for years. Why would I do this? There's several reasons.

• Speed of development. It's pretty fast to hack something up in Python.
• It helps with one of the project's major goals: be easily modifiable. Python is widely understood and considered unintimidiating. If I wrote it in a manual memory management language some would (probably quite rightly) feel concerned that if they modified it they'd introduce some variety of security error. It also has a nice tweaking experience: just open the damn file and edit it and run the program, no needing to set up a complex dev environment.
• It's just a wiki, I mean come on.

I'm serious about that last one. A wiki is a simple CRUD app that takes a markup and renders static pages. It should be largely I/O-bound. Python is computationally slow but what computations is this thing going to even be doing?

I think that while yes a lot of Modesty™ can be found in choosing the right programming language, I think you can also achieve a lot just via careful programming, and mindful solution choices. People don't check their memory usage, people reach for large complex solutions to problems that they just don't have. I want to test the idea that people don't have to change absolutely everything about the way they program to end up with better software. I'm going for modest not optimal.

A thing I think more people should care about though is choice of dependencies. Speaking of which...

You should consider SQLite for your website

For those of you unaware, let me introduce you to SQLite. SQLite is the most deployed SQL DB, estimated at over a trillion instances. It's used in aeroplanes and is tested to that industry's standard. The devs care about it's size, speed, backwards compatibility, and reliability. They've pledged compatibility until the year 2050. Python comes with bindings for it in the standard library. And it will be the storage mechanism for Mosiki.

It's considered an unorthodox choice for a webserver DB but that's actually not outside of it's goals; the SQLite website itself uses SQLite as its backing, and cites it being a good choice for small-to-medium traffic servers. So should be perfect for my target use case. It also should lower the spec requirements a lot compared to using something like Postgres.

Additionally, SQLite is actually quite fast. DokuWiki just uses files but SQLite will probably be faster than I ever will be at that. And what may come as a shock: SQLite is actually faster than typical DBs, at least for simple cases with low write-concurrency.

(Source: https://sj14.gitlab.io/post/2018/12-22-dbbench/)

By turning on one setting, Write-Ahead Log mode, SQLite outshines popular web server dbs for simple cases. And this really shouldn't be that suprising because it's just doing less work than typical DBs. There's no server, no seperate process, it's an API that reads and writes data to a file in an atomic ACID-conforming way. And yes it won't handle the same sort of concurrent writer load as Postgres but I don't have that problem, and neither do many websites.

If there's anything you take away from this post, it should be that you should consider SQLite a real option for web-servers.

Markdown

So I need a markup for Mosiki. I could invent one. But honestly, why? Markdown is pretty good. It's pretty simple. And there's highly tested implementations. markdown-it-py is the one you're meant to use for Python these days. It has some HTML safety features, they've benchmarked it against other solutions, it is even extensible. There's a footnote extension, and I'll definitely use that. Overall, seems like a reasonable solution that even allows for some expansion in the future. I am trying to be careful about dependencies. I don't want to accumulate 1000s of dependencies NPM-style. But this seems like a reasonable choice, and worst-case scenario it could always be replaced with an even lighter Markdown parser.

The troubles with security

An area of conflict I wasn't expecting for this project was between Modesty and Security. I decided to look into state of the art password encryption and found that part of it now is deliberately high memory usage. That sounds weird at first but it makes sense on further pondering. Computers these days can do a whole lot of work in parallel, particularly on GPUs. If you want to make sure that your encryption method is safe from large scale brute force attacks you want to make sure it's hard to defeat through all our new parallelized programming power.

And the logical way to do that is to simply use memory. Parallelization has improved much faster than memory sizes. Argon2 is seemingly the state of the art these days. And it's primary recommended setting uses 2GB of RAM. Good luck launching a parallelized brute-force attack against that. It does have a low-memory recommendation that uses 64MB but... honestly that still feels a bit much? I'm trying to target 512MB VPSs here, so 64MB is a lot. But I'm not a security expert, and so I probably shouldn't tamper. I can make a lock and limit it to verifying one login at a time which should make it fine for my use case.

Flask? Maybe?

Flask is a webframework for Python that clearly had some amount of Modesty™ in it's design. It's not trying to do everything but rather be an alright starting point and let people solve the rest of it with extensions if they want. It comes with a templating library, a request life-cycle, routing, some logging stuff, and really not much else. It has about 17k lines to it so it's far from the worst. Strictly speaking I think I could likely not use this and go for something a even more barebones, but it seems like a reasonable-enough starting point. Django is the "industry standard" Python web framework and is huge. ~400K lines to Flask's ~17K. Bottle is another one, quite like Flask to my understanding, and even smaller, has no dependencies. But is fairly niche and doesn't seem the most actively developed. Quart was another Flask-like that is smaller than the original and also uses asyncio (more on that later) but it's also in very low development. I'll go with Flask for now, but moving to Bottle or Quart could make sense.

Gunicorn? Gevent? Eventlet? Stackless? Reverse proxies?

The Python ecosystem has a lot of web server options and even an entire web server interface standard called WSGI. I have been doing a lot of reading and comparing options. Flask comes with a development server, but it's not seriously tested as a production HTTP solution and so you can't complain if it has some obscure vulnerabilities and problems.

A common setup seems to be Nginx->Gunicorn->Flask. Nginx acts as a reverse-proxy: a server that will act as a bouncer. It's highly tested for security and will provide buffering and other features that can be used to improve stability. I can find benchmarks of it handling 10 thousand concurrent connections with 18 MB of RAM so it should be light enough for my case. I was tempted by lighttpd but I decided to go with the more known thing. Gunicorn provides a worker-model and some supervision but honestly, I probably will be single threaded and don't care about the worker model, so screw it, I won't use Gunicorn. I'm not hugely convinced it'll add much other than complexity.

Additionally, Python has a number of things to help with doing a crap tonne of requests. Particularly they make requests asynchronous via things like coroutines or green-threads, basically light-weight alternatives to threads that you can scale up into the thousands or even tens of thousands. There's a lot of options in this space, Stackless, Gevent, Eventlet, Asyncio. Asyncio is in the standard library and will likely be the popular one in the long term.

Now, I don't really plan to have 1000s of users at anytime for Mosiki, but I think it's good to try and be reliable. Additionally, I'm hoping these systems will help prevent stalls caused by slow I/O requests from one user impacting every other user. For now, I will try using Gevent because it's quite easy to add to a flask program and Eventlet didn't want to work for whatever reason so I can't really compare.

Now, perhaps one of the technologies I have chosen to use is a bad bet. But the beauty of making a Modest Program is that you try to keep the code so simple that anyone with familiarity with the programming language can quite easily maintain it themselves. If I've made a poor choice for Mosiki, it should be easy to change later.

Testing time

So my flask+Gevent server with a WIP Markdown page renderer pulling pages from SQLite consumes 35 MB at idle. Combined with nginx doing its 18MB from that benchmark earlier and let's assume Argon2 is using that 64MB setting. That's 117 MB so far on the server stack. And over half of it simply reserved for security reasons.

Alpine Linux is a Linux distribution designed to be tiny and secure. It's based around the impressive musl and busybox projects (I'll let you research those if you feel like it). At boot it uses 41 MB in a virtual machine, at least it does if you're in text only mode. If you're trying to go for an extremely cheap server, using such a distribution helps. For the sake of safety, let's double it's memory usage to 82 MB for my equation. 82+117=199 MB of RAM. That does feel a little on the high side of what is likely hardware-optimal, but let's see how it scales under load simulation. I chose to use hey as my benchmark tool for no particular reason.

20 Concurrent Users doing 2000 (total) simple page loads

Summary:
  Total:	3.1373 secs
  Slowest:	0.0536 secs
  Fastest:	0.0118 secs
  Average:	0.0312 secs
  Requests/sec:	637.4907
  
  Total data:	2452000 bytes
  Size/request:	1226 bytes

Response time histogram:
  0.012 [1]	|
  0.016 [1]	|
  0.020 [2]	|
  0.024 [2]	|
  0.029 [2]	|
  0.033 [1650]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.037 [246]	|■■■■■■
  0.041 [58]	|■
  0.045 [24]	|■
  0.049 [11]	|
  0.054 [3]	|


Latency distribution:
  10% in 0.0291 secs
  25% in 0.0296 secs
  50% in 0.0304 secs
  75% in 0.0315 secs
  90% in 0.0345 secs
  95% in 0.0368 secs
  99% in 0.0438 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0000 secs, 0.0118 secs, 0.0536 secs
  DNS-lookup:	0.0000 secs, 0.0000 secs, 0.0003 secs
  req write:	0.0000 secs, 0.0000 secs, 0.0002 secs
  resp wait:	0.0311 secs, 0.0115 secs, 0.0533 secs
  resp read:	0.0000 secs, 0.0000 secs, 0.0003 secs

Status code distribution:
  [200]	2000 responses

So, at 20 concurrent users, probably quite high for the intended use case, this simple page load is basically instant by internet standards. I saw no real perceptible memory increase. Now, let's try a more proper stress test.

1000 Concurrent Users doing 2000 (total) simple page loads

Summary:
  Total:	7.4600 secs
  Slowest:	7.3279 secs
  Fastest:	0.0631 secs
  Average:	1.3052 secs
  Requests/sec:	268.0980
  
  Total data:	2452000 bytes
  Size/request:	1226 bytes

Response time histogram:
  0.063 [1]	|
  0.790 [1016]	|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  1.516 [303]	|■■■■■■■■■■■■
  2.243 [310]	|■■■■■■■■■■■■
  2.969 [41]	|■■
  3.695 [271]	|■■■■■■■■■■■
  4.422 [41]	|■■
  5.148 [0]	|
  5.875 [0]	|
  6.601 [0]	|
  7.328 [17]	|■


Latency distribution:
  10% in 0.2357 secs
  25% in 0.3473 secs
  50% in 0.5573 secs
  75% in 1.9608 secs
  90% in 3.3497 secs
  95% in 3.5311 secs
  99% in 3.7537 secs

Details (average, fastest, slowest):
  DNS+dialup:	0.0201 secs, 0.0631 secs, 7.3279 secs
  DNS-lookup:	0.0028 secs, 0.0000 secs, 0.0582 secs
  req write:	0.0021 secs, 0.0000 secs, 0.0471 secs
  resp wait:	1.2688 secs, 0.0110 secs, 7.3278 secs
  resp read:	0.0001 secs, 0.0000 secs, 0.0025 secs

Status code distribution:
  [200]	2000 responses

Obviously the average load speed here gets a bit bad, but it handled it, didn't max out a CPU core, and most importantly, memory usage didn't go nuts. Nginx seemed to end up only using about 15 MB, and my flask+Gevent server got to 36 MB, 1MB greater than idle. Setting aside some memory for Argon2 and my Alpine ballpark with it's extra sapce, that's 197MB of RAM for what is likely a very high load scenario for what I'm making. Scaling well so far then. That 1 MB increase conveniently is around about the number of current users times the payload size, I suspect that is not an accident. If I keep my page sizes small, via tightly curated CSS, no JS, and sensible image formats such as PNG, then this all should scale quite well.

Weirdly, the peak memory usage according to grep ^VmPeak /proc/pid/status of the flask+Gevent server was around 215 MB but it has that peak seemingly immediately. I know Gevent does monkey patching on the standard library to work, and maybe that causes an initial memory surge, which sucks, but it's probably managable. The server is possibly still usable on a 256 MB RAM VPS with some Swap?

Point is though, memory usage seems low and so far the Python bit is only 36MB. You can only really improve the RAM substantially by using an even tinier OS, or compromising on password security. All of the Modesty so far has been about careful choices in what I'll build on; trying to add things that seem like good bang-for-buck and avoid things that seem like too much for my problem. I did a lot of looking around online for benchmarks where possible. So far, I think it's working out quite alright.

Goodbye for now

My work on Mosiki is no where near done; it's basically a simple markdown-from-sqlite renderer at the moment. And as I flesh it out more I'll be able to make tests with more representative pages. In theory you could benchmark an edit work load, but honestly I don't think that's likely very relevant to my use case? I'll probably do it anyway for completeness sake.

The next post I do in this series I'll show off where Mosiki is at feature wise, go over some random engineering decisions I made and some actual code. It'll probably be a post less relevant to the performance goals of Mosiki.

Thanks again for reading. If you have any comments, questions, or suggestions feel free to reach out.

I've heard it, you've heard it, there's absolutely no news in saying "WOOOW computer hardware sure is impressive these days! You can put a super-computer in your pocket!"

Sadly, an observation not quite as common is "and yet everything kinda sucks to use." It's true that common smartphones are incredible machines by the standards of computer history and much more powerful than home desktop computers were in the 90s. But there's a growing sentiment amoungst many variety of whiney programmers— which I definitely am by the way— that our field squanders this potential.

Don't mind me, just gonna make a sweeping statement based mostly on gut feel and anecdote, no biggie

I'm going to make a somewhat bold statement: I personally do not see any reason for the average consumer, and even most enthusiasts, to have computer hardware seriously improve from this point. I even feel that had our industry tried harder, computers from ten years ago would have easily fufilled the majority of needs. There are definitely exceptions to this such as video editors, but again they are exceptions.

Conceptually, when you think about the size of 16GB of RAM.... is a lot of information. That's 4 billion 32 bit numbers. If a common user workload can seriously approach that amount of data I feel like something is up.

And this isn't just pointless complaining— although that is one of my favourite passtimes. No, this is quite impactful. The world is effectively wasting large amounts of energy, industrial capacity, rare minerals, consumers' time and money on what really feels like nothing; we upgrade our computers and yet we don't have a snappier experience than we did 10 years ago. They say we do this because hardware is cheap and programmer time is expensive, but I'm curious if the cost of that hardware really factors in all the lost-opportunity cost and negative externalities. We don't have an endless supply of minerals, and hell, we're even in a silicon chip shortage right now. Additionally, this is often a negative externality: the cost is on the user's hardware. It's easy to say it's cheap when you don't pay for it.

Now, I could just sit here and complain. After all, what's the fun of a blog if not engaging in grossly oversimplified criticisms. But I think if we want things to change, we need to participate in and foster a culture of trying. I'm not gonna say absolutely everything must be done in the most optimized way ever. But I think it'd be good to begin a culture of checking these things, paying attention to our software's resource usage and asking "Does that seem reasonable?". I think that mindset alone would go a long way. Here on this blog I'm going to call this "Modest Programming": simply asking yourself if your program is reasonable in what it asks of the user's computer, and really trying at all.

This idea is far from unique. We've got suckless, we've got permacomputing, we've got uxn. I could name more. I don't claim to have invented this notion, and I'm not pitching "Modest Programming" as the term that should be used around the net. It's just the wording I like, and so I'll use it on this blog.

But enough with that. It's time to practice what I preach. Emphasis on practice.

The Experiment

Recently I was talking in a discord server that I frequent, and commonly exchange information and opinions about deeply obscure japanese video games. In case you had not figured it out so far: yes, I am in fact a filthy nerd.

There's a question that gets asked there reasonably often, that has no simple answer, and it raised to my mind the thought of "Man, it'd be kind of nice to have a community wiki for some of that stuff." Now, obviously random free wiki hosts like Fandom™ (formerly Wikia®) exist, but personally, I like owning community spaces. And setting up your own sites is fun. So I found myself looking into what wiki software is out there, and how cheap I could reasonably run a small community wiki.

Sadly, I couldn't really get much of a solid answer on how low-spec existing wiki software went. I'm sure they're not insane but if I could get this thing running very reliably on a 512 MB RAM VPS I'd very much appreciate that. And looking around it felt like none of the many perfectly high quality solutions had really been paying attention to this particular point. There is DokuWiki and don't get me wrong. It looks pretty solid. It doesn't even use a DB which I think is a good choice for both being light-weight and simple; the qualities someone would want for a small community wiki. But again, I couldn't really find exact details on just how low I could turn this thing down, and how well it'd handle it. Generally using little memory is one thing, but can it resist collapse in the face of some reasonable traffic?

Now, I haven't done too much web server programming in a wee while, and I felt like I should get some practice. Also, I'd like to give myself a Modest Programming Challenge™. So, why not try making a modest wiki implementation? A wiki is not the most impressive thing ever, and so you may think that that makes them not a great example for Modest Programming. I disagree. I think because wikis are so simple in their general function it should be reasonable to run one on some innanely small devices.

I will also admit, and I'm sorry if there's any PHP fans reading this, but a very large portion of established wiki server options use PHP, and quite frankly. I'm just prejudiced against any software built on that technology. I don't want to run it, I don't want to administer it. I'm absolutely sure you can point to perfectly good examples of PHP server software. Even wikipedia is one of them. But I've read too many things to feel comfortable touching it. Forgive me for my paranoia.

The product of this will have reasonably little use case. DokuWiki is likely a generally good solution for the small community case, and obviously MediaWiki is extremely battle-tested. But this is partially for practice, fun, and there are a few things I'd like to do different.

Speaking of which, let's lay out some goals.

Goal 1: Should run on extremely cheap VPSs

The specific use case I'm targeting is moderately technically capable people who want to quickly set up a small community wiki at extremely low cost. This site is hosted on Neocities so you can probably guess that I'm somewhat opposed to the "BigTech" CorpoWeb™. I think if we want more community-owned spaces, we need to make them as cheap to run as possible.

Additionally, as I have already stated. A wiki really is an extremely simple app in terms of what it actually does, sure it's not technically a static site but it's very close. I feel it should be possible to run on pathetic hardware.

Goal 2: Should be very simple to administer

Again, community-owned spaces online run largely on volunteer labour so we need to respect that labour. I think the general set of things someone should need to learn to use this and administer it long term is:

• What a VPS is and how to set one up
• Some bash
• Maybe some SQL
• An extremely simple configuration setup for the wiki settings

The last one is fairly important. Most small communities don't really need complex access rules. They want a wiki that is easy to set up and then gets out of their way.

Goal 3: Should be very easy to modify

Now, this one is something that DokuWiki does not satisfy in my opinion. I am a big believer in simple software for technical users who can come to understand the entire thing in quite some detail. I use a custom modification of makesite to generate this blog (okay, it's not the most impressive blog, I will admit) and I love the solution. It's a dirt simple python script with a template that generates a static site. If it doesn't do something you want, you can make it do it because it's so simple that you can read through the entire thing and its templates in a lunch break. And honestly, for a reasonably technical person that might be a better solution for customizability sometimes. Contrast trying to make a complex piece of software that can do everything as long as you want to invest the time in figuring out how you make it do anything. Then you pray to god that it doesn't surprise you with anything weird at some point that you can't debug because it's far too complex to just solve trivially.

So that's a thing I want for this modest wiki. DokuWiki doesn't fit the bill because according to OpenHub it's 233K lines of code. That's not a tool you can understand entirely.

Goal 4: Should deliver trivially small payloads, fast

I'm going to layer more stereotypes on to myself here and say that the web is a goddamn mess. Pages are unreasonably huge for most purposes. Things take way too long to load. I use Google's Pagespeed Insights to evaluate performance of my blog and I'm happy that it's a solid 100. It could be even more streamlined; most of my website's by filesize is fonts when I could just use fonts that come with most operating systems. But it's still pretty good.

DokuWiki does not perform well on this metric for phones. Additionally a very simple page comes in at 1.35MB and I'm just unimpressed. 1.15MB of that is just Javascript. Wikipedia does better with 526KB on a reasonably simple page, one with pictures even. But 309KB of it is Javascript.

I'm not some hard-core NoScript kinda guy but I have to ask: what on earth are these pages doing with so much javascript? They're server-rendered pages. Why do they even use Javascript at all? I can understand why a wiki may have them in editor pages or admin pages, but for viewing a page? Why? This is what I mean by software needs to be more modest; try asking why you are introducing some requirement, whether it be on hardware, internet connections, or just some additional bit of technology. In this case, I can't really see much of a justification.

Now I might end up changing course on this but I'm going try for zero javascript. There are definitely useful things you can do with JS on the editor side, but fundamentally, I don't see a hard need. So why not? And it makes this project potentially more useful for someone. Sure that someone is NoScript weirdos, but I am a weirdo. Not that exact kind, but us weirdos gotta stick up for each other.