Juho Snellman's Weblog

I don't want no 'wantarray'

jsnell@iki.fi — Tue, 18 Jul 2017 18:00:00 GMT

A while back, I got a bug report for json-to-multicsv. The user was getting the following error for any input file, including the one used as an example in the documentation:

    , or } expected while parsing object/hash, at character offset 2 (before "n")

The full facts of the matter were:

The JSON parser was failing on the third character of the file.
That was also the end of the first line in the file. (I.e. the first line of the JSON file contained just the opening bracket).
The user was running it on Windows.
The same input file worked fine for me on Linux.

Now, there's an obvious root cause here. It's almost impossible not to blame this on Windows using CR-LF line endings, where Unix uses just LF. The pattern match is irresistible: works on Linux, fails on Windows, fails at the end of the first line. And I almost answered the email based on this assumption.

Except... Something feels off with that theory. What would be the root cause here? "Wow, I can't believe that the JSON spec missed specifying the CR as whitespace"? No, that makes no sense, nobody would define a text-based file format that sloppily. 0

How about: "Wow, I can't believe the JSON module of a major programming language has a bug making it fail on all inputs on a major operating system, and it took a decade for anyone to notice". That doesn't seem plausible either.

So I tried to reproduce the problem, by making a file with DOS line endings and running it through the script on Linux. That worked fine. Hm. Put in some invalid garbage, and you get a parser error as expected. Double-hm. But the error message I got was very different from that in the bug report. Could it be that it's using a totally different JSON module altogether?

Turns out that's basically what was going on. Perl's JSON module doesn't actually do any parsing itself. It's mostly a shim layer, the actual work is done by one of several different parser modules. On Linux, I'd been getting JSON::XS as the backend (XS is Perl-talk for "native code"). In cases where JSON::XS is not available, the shim module would use a pure Perl fallback, e.g. JSON::PP.

Ok, so force the JSON module to dispatch to JSON::PP. Success! Problem reproduced. Guess it really was buggy parser after all. Remove the DOS line endings, just to be sure... And it's still failing. WTF?

A bit more digging revealed that the error message was actually a lie. The problem wasn't with the whitespace, but with there being an end of file right after said whitespace. The input to JSON::PP contained just a single line, not the whole file! At that point, the actual problem becomes obvious and the fix trivial:

-    my $json = decode_json read_file $file;
+    my $json = decode_json scalar read_file $file;

I was using the read_file function from File::Slurp to read the contents of the file. Unfortunately that function behaves differently in scalar and list contexts. In scalar context, it returns the contents of the file in a single string. In list context, an array of strings. What had to be happening was that the context was changing based on the backend.

And just why would changing the parser backend change the context for that read_file call? As it happens, the JSON module does not actually define decode_json, but directly aliases to the matching function in the backend. For example:

*{"JSON::decode_json"} = &{"JSON::XS::decode_json"};

JSON::XS declares the function with a $ prototype forcing the argument to be evaluated in scalar context. JSON::PP uses no prototype and thus the arguments defaulted to being evaluated in list context.

The blame game

So, that's the bug. But what was the real culprit? I could come up with the following suspects.

Me, for using File::Slurp for this in the first place. "Oh, I just always pass a file-handle to decode_json" said one coworker when I described this bug. And that would indeed have side-stepped the problem, and read_file is just saving a couple of lines of code. But it's exactly the couple of lines of code I don't want to be writing: pairing up file opens/closes, and boilerplate error handling.
Me, for not realizing that the code was only working by accident. I knew read_file works differently in scalar and list contexts. I also knew this case needed scalar context, and had no special reason to believe that decode_json would provide it. The default assumption should have bene for this code not to work. When it did, I should not have accepted it, but figured out why it worked and whether it was guaranteed to work in the future.
The JSON module, for not explicitly documenting the inconsistent prototypes as part of the interface. I don't know that anyone would actually notice that in the documentation though. It might end up as just cover-your-ass documentation.
The JSON module, for directly exposing the backend functions with aliasing, for a minimal performance gain. It's a shim: isn't the whole point to hide away the implementation differences from the user?
The File::Slurp module, for using wantarray to switch behavior of read_file based on the context.
Perl for having the concept of different contexts in the first place.
Perl for allowing random library code to detect different contexts via wantarray.

The thing that really sticks out to me here is overloading of File::Slurp::read_file based on the context. Returning a file as a single string vs. an array of lines are very different operations. There is absolutely no reason for them to share a name. It'd be simpler to implement, simpler to use, and simpler to document. It's even already in a library, so it's not like there would be any kind of namespace pollution by using different names. (Unlike for the uses of context-sensitive overloading in core Perl. Sure, count would probably make more sense than scalar grep. But it would be a new name in the global namespace).

What about wantarray? It's what's enabling this bogus overloading in the first place. I've been using Perl for 20 years, writing some pretty hairy stuff. As far as I can remember, I haven't used wantarray once. And what's more, I don't remember ever using a library that used it to good effect. The reason context-sensitivity works in core Perl is the limited set of operations. One can reasonably learn the entire set of context-sensitive operations, and their (sometimes surprising) behavior. It's a lot less reasonable to expect people to learn this for arbitrary amounts of user code.

It's a bit unfortunate that function aliasing can cause action at a distance like this. But at least that's a feature with solid use cases.

So I think that's where I fall on this. It's all because of a horrible and mostly unnecessary language feature, used for particularly bad effect in a library. It feels like avoiding this kind of problem on the consumer side is almost impossible; it'd just require superhuman levels of attention to detail. Avoiding it on the producer side is really easy: wantarray: just say no.

Footnotes

[0] Did you nod and agree at "that makes no sense"? Haha. The original JSON spec does say that "whitespace can be inserted between any two tokens", but doesn't actually define whitespace.

json-to-multicsv - Convert hierarchical JSON to multiple CSV files

jsnell@iki.fi — Tue, 12 Jan 2016 14:30:00 GMT

Introduction

json-to-multicsv is a little program to convert a JSON file to one or more CSV files in a way that preserves the hierarchical structure of nested objects and lists. It's the kind of dime a dozen data munging tool that's too trivial to talk about, but I'll write a bit anyway for a couple of reasons.

The first one is that I spent an hour looking for an existing tool that did this and didn't find one. Lots of converters to other formats, all of which seem to assume the JSON is effectively going to be a list of records, but none that supported arbitrary nesting. Did I just somehow manage to miss all the good ones? Or is this truly something that nobody has ever needed to do?

Second, this is as good an excuse as any to start talking a bit about some patterns in how command line programs get told what to do (I'd use the word "configured", except that's not quite right).

What and why?

I needed to produce some data for someone else to analyze, but the statistics package they were using could not import JSON files with any non-trivial structure. Instead the data needed to be provided as multiple CSV files that can be joined together by the appropriate columns.

As a simplified example, instead of this:

{
  "item 1": {
    "title": "The First Item",
    "genres": ["sci-fi", "adventure"],
    "rating": {
      "mean": 9.5,
      "votes": 190
     }
  },
  "item 2": {
    "title": "The Second Item",
    "genres": ["history", "economics"],
    "rating": {
      "mean": 7.4,
      "votes": 865
   },
   "sales": [
     { "count": 76, "country": "us" },
     { "count": 13, "country": "de" },
     { "count": 4, "country": "fi" }
   ]
  }
}

My "customer" needed this:

item.csv

item._key	item.rating.mean	item.rating.votes	item.title
"item 1"	9.5	190	"The First Item"
"item 2"	7.4	865	"The Second Item"

item.genres.csv

genres	item._key	item.genres._key
sci-fi	"item 1"	1
adventure	"item 1"	2
history	"item 2"	1
economics	"item 2"	2

item.sales.csv

item._key	item.sales._key	sales.count	sales.country
"item 2"	1	76	us
"item 2"	2	13	de
"item 2"	3	4	fi

One way to do this would have been to just change the program I used to produce the output. That would have been a bit annoying since the CSV output codepath would have been basically completely separate from the JSON one (which was basically just a JSON::encode_json on the natural data structure. It's almost easier to just have a generic converter than one specific for that one app (the documentation is as long as the program itself). The only question is how to configure the generic mechanism for the specific case.

How command line tools get run

Could this "just work" out of the box with no settings at all? Not really, there's multiple ways of interpreting the data. A compound value could mean either the addition of more columns (ratings in the example) or adding rows to another CSV file (sales in the example). Consistently choosing the first interpretation would not work at all, while in the latter case you'd get really awkward entity-attribute-value-style output.

Ok, so some configuration is needed. What kind of options do we have for doing that? Command line flags tend to be the simplest to start with, though they'll often eventually become complex either by developing ordering dependencies between flags (to express different semantics) or by the values developing some kind of complicated internal structure.

Both of those actually happen for this tool. To run it, you need to pass in multiple --path command line options, each containing a pair of a patterns and the action to take for values whose path matches the pattern. (Just the first matching action is taken). For the above example those flags were:

   --path /:table:item
   --path /*/rating:column
   --path /*/sales:table:sales
   --path /*/genres:table:genres

Scalar values have an automatic fallback handler that just outputs the value as a column, but for compound data fields not finding a match is an error. In these cases the error message will print out some suggestions on what command line arguments could be added to resolve the error, for example:

Don't know how to handle object at /*/appendix/. Suggestions:
 --path /*/appendix/:table:name
 --path /*/appendix/:column
 --path /*/appendix/:row
 --path /*/appendix/:ignore

The next option would be feeding some kind of a schema file to the tool, which would then be used to guide the process. For example if the schema says that a type of object has a static set of fields, those fields are probably columns. If it has an unknown set of keys, it's probably more like tabular data.

The problem is that writing the schema would be a bit of a pain, and it would be much harder for the conversion tool to guide the user through an iterative process of getting the schema definition right. One could maybe generate a schema file from the data file itself, and edit any bits that the autodetection goes wrong. Schema generators do exist, for example jsonschema.net, but at least that one doesn't have enough knobs to tweak to even get this basic example right. And the mistakes are such that fixing them would take a fair bit of work. Reliable automated schema generation would make for some pretty epic yak shaving in the context of this tiny tool.

Maybe if people really did write JSON schemas for everything it would make sense to use that existing infrastructure. But I've never seen one of those in the wild, the spec is complicated, and JSON schemas are not particularly well suited to this use case. (Really you'd want a custom schema format, but then it's completely guaranteed that there's no pre-existing schema file to use).

And here's the thing... It's not just this specific case. It never feels like any kind of declarative schema is the right solution. In a couple of decades of writing data munging scripts I can remember just a single case of basing the solution on an external description of the data. And that single exception had several people working on the tool full time. Sure, it's great to have a schema of some sort for for your data interchange or storage format, for use in validation, code generation, automated generation of example data, or other things like that. But for actually processing it? It's just an incredibly rare pattern.

And finally, could this be a use case for a special purpose language? If schemas feel like a rarity, little languages are the opposite. Especially in classic Unix they are ubiquitous.

As a recovering programming language addict, I have to be deeply suspicious every time a new language looks like the right solution. Is it really? Or is this just an excuse to fall off the wagon again, and implement a language. (Not a big language, man. Just a little one, to take the edge off).

It's also clear that the general idea of a JSON processing language is solid. Some already exist (e.g. jq), but there could be room for multiple approaches. Writing sample programs to see what a language for JSON processing and transformation might look like was a fun way to spend a couple of hours on the boring "no internet" leg of a train journey. ("It could have this awk-like structure of a toplevel pattern matching clauses, but on paths instead of rows of text, and with a recursive main loop instead of a streaming one, and and and...").

If I kind of wanted to write this, the idea is good, and an initial implementation is not an unreasonable amount of work, why not do it? Well, even if a script written in this hypothetical language to translate from hierarchical to tabular data would have been pretty simple, it would still have been a program that the user of the tool needs to write in a dodgy DSL. And since the language would have been much more generic than a mere conversion tool, it it would also have been impossible to guide the user through a process of iteratively building the right configuration (like is now done via the error messages).

In all likelihood it'd mean that nobody else would ever use the tool for the original purpose. The less powerful and less flexible version is just going to be more useful purely due to simplicity.

So sanity prevailed this time. But tune in for the next post for an earlier example of where my self control failed.

Command languages as game user interfaces

jsnell@iki.fi — Mon, 08 Dec 2014 12:00:00 GMT

In the previous post in this series, I promised to discuss in detail some of the positive and negative consequences of the less conventional design choices of my online Terra Mystica implementation. If you have no idea of what that is, reading at least the intro of that post might be a good idea. This post will just deal with one design choice, but it's the elephant in the room: the command language.

The canonical internal representation of a game in my TM implementation is as a sequence of rows, each describing a some number of player actions specified in an ad hoc mini language, or administrative commands that change the game setup in some way (for example setting game options, or dropping a player from the game partway through). This is what it might look like:

yetis: action ACT4
cultists: upgrade E6 to TE
cultists: +FAV6
giants: Leech 3 from cultists
giants: pass BON4
yetis: Leech 2 from cultists
cultists: +WATER
dragonlords: Decline 2 from cultists
dragonlords: dig 1. build G6
yetis: send p to EARTH
cultists: action FAV6. +AIR
dragonlords: pass BON7
yetis: upgrade E7 to TE. +FAV11
giants: Leech 3 from yetis
dragonlords: Leech 2 from yetis
cultists: Leech 2 from yetis

That's a short excerpt from the middle of a random game. A full game generally runs for about 400 rows.

What do I mean by this being the canonical internal representation? Only a few parts of the game state are actually persisted separately in the DB; these are things that might almost qualify as metadata, such as whose turn is it to move, is the game still running, and what were the final rankings of a finished game. But in general the only way to find out the current state of the game is to evaluate the whole sequence of commands from start to finish. This is in fact done for almost every operation on the site (viewing a game, previewing a move, saving a move, viewing the or editing the game in an admin mode, and so on).

In addition to being the canonical internal representation, the command language is also the canonical user interface; the fundamental operation players do is enter new rows into the command sequence. Often this is done by writing the commands manually, though there are GUI shortcuts of one form or another available for almost all operations.

This might sound like a slightly insane way of doing things, but it does have some benefits as well. I've made several digital board game adaptations of varying levels of completeness over the years, used tens of other ones, and this solution hits the closest to my personal sweetspot.

A taxonomical diversion

Before discussing the fallout of this design decision in more detail, it's probably useful to do a quick tour of some of the main axes in the design space. (I'm of course just describing the extremes, while in the real world most examples would fall on a continuum).

First, there's the question of the interaction model which might be abstract or skeuomorphic. In a skeuomorphic design the player doing input on a computer would still be mimicking the actions of someone playing the game with physical pieces and no computer assistance.

In an abstract design the player would only input the parts of the move that are necessary to uniquely distinguish it from other possible moves, with any bookkeeping and mandatory intermediate steps being carried out automatically. Likewise in a skeuomorphic design the software provides information through the same methods as the original physical game, while an abstract design will automate some of the mechanical parsing of the game state. Or even just the question of using the graphical assets of the original game, generally optimized for sales, versus using digital-first assets optimized for clarity.

As an example of this axis, in the 18xx series of games a substantial amount of playtime is spent computing the exact routes of a number of trains on a complex rail network. I'm aware of three solutions that are actually in use, and there is a fourth plausible one, in order from least to most abstract:

The user manually decides on the routes, computes their values with no computer assistance, and those values are used with no validation. Examples: ps18xx, early versions of Rails.
The user enters valid routes through a user interface. The software computes the values of the routes, and distributes the income from the company appropriately. Example: rr18xx.
In games with requirements that all routes must be optimal, the software could compute an optimal route but only for the purpose of rejecting any manually computed unoptimal ones. Examples: None. (Though it's similar to what's done in the SlothNinja implementation of Indonesia, a game that probably counts as an honorary 18xx)
The software automatically finds an optimal set of routes and computes their values. Examples: The ancient DOS-based 1830 from Simtex, recent versions of Rails.

My own tastes run toward maximum abstraction, I've rarely if ever seen a digital boardgame conversion that needed to be more skeuomorphic. But this is not a universal view. There are definitely people who will refuse to play a conversion that does not use the same graphics as the physical version. Or who will strenuously argue against automatic finding of optimal routes in 18xx, on the basis that being evaluating routes is a core skill in the game when making decision about route building, and that skill can only be acquired by getting sufficient practice in manual route computation.

A second axis is the internal representation, which could be based on either log replay or stored state. In a log replay system the game is stored as a series of steps from the starting setup to the current state. In a stored state system the game is stored as the current values of all pieces of the game. How much money does every player have, which round is it right now, what's in this exact space on the map, and so on.

A third axis is the input model. Moves could be entered either through direct or indirect manipulation. In a system using direct manipulation, the player would for example see a graphical display a map and be able to click or drag on a unit to enter a move for it. In an indirect system the player observes the game state in one place, and enters their moves using some completely unrelated system.

I think most digital boardgames use a direct input model, but there are also a fair number that have a menu-driven system of some sort. The only examples I know of that go a bit further with indirection by providing a command language are my ancient Paths of Glory mapper and the even older Diplomacy PBEM judges. If you have other examples, I'd love to hear of them.

Direct manipulation is often, but not always, linked to excessive skeuomorphism in the interaction model. For example I find it almost painful to play most Vassal modules, with their hyper-direct interaction model of dragging and dropping counters around, manually drawing cards from a deck or rolling dice. Digital boardgames are not the same media as physical boardgames, and should play to their unique strengths. But these are in fact orthogonal concerns, and there's no reason for why a direct manipulation model couldn't also provide useful input and computational abstractions.

Whew, so much for the theory. In this taxonomy Online Terra Mystica is pretty far toward the abstract end, and is fully in the log replay camp. While it has a half-hearted attempt at adding some direct manipulation concepts to the UI, it started off as an indirect system and deep inside that's what it is. It also chooses to merge the input format and the log format into one entity. So what does this mean?

Feature set

Perhaps the signature feature of the site is the planner. This tool allows the player to enter an arbitrarily long sequence of actions - all the way to the end of the game - and see what the effects would be. Are all the moves valid? Are there sufficient resources available to do all of this? Oh, I don't have enough resources? Well what if I do this on round 5, and delay that action to round 6. In cases where the plan fundamentally depends on the opponents doing something, it's possible for the plan to also contain arbitrary resource adjustments. And finally, since the command language supports comments, these plans can be properly documented so that when you return to them in a day or two, you can remember why you wanted to do these particular moves.

I think this feature is intrinsically linked to the command language as a user interface, and it might actually be unique. There are some games with other kinds of interfaces that allow you to play the game forward, and then undo / rewind / reload. But simply being able to play the game forward is not sufficient to make this a useful tool. It's only the ease of inserting, reordering and deleting moves that makes it possible to use this as a matter of course, rather than only under the most exceptional circumstances.

A somewhat related feature is undo. Inflexibility in allowing moves to be taken back is the bane of many forms of digital boardgames. When playing a game face to face, most groups will generally allow at least some level of taking back moves. In some cases all moves are final immediately (this has always been the primary problem of the otherwise brilliant implementation of Brass at Order of the Hammer). In some other implementations there are distinct checkpoints, for example BGO's Through the Ages allows undoing back to the start of your full turn, but no other rollbacks (clicking 'finish turn' is final, as is any kind of action during an auction or war resolution). These two are, I believe, examples of undo being limited for design reasons. At rr18xx meanwhile rollbacks are possible until the previous action of each player. Here my understanding is that the overriding issue is technical, as the rollback is essentially a full restore to a previous database snapshot, and there are resource constraints on how many snapshots can be kept.

The solution Online TM takes to this is to grant the creator of the game arbitrary powers to edit the history at will, the admin mode. Not only can they undo the last move or couple of moves. If there was a mistake made three moves back, they can go and fix it (and they can fix it without forcing the intervening moves to be redone). This feature is fully tied to a log replay mode of operation. While more limited forms of undoing could be implemented as a reverse log replay from the end state or through state snapshots, this more complete form depends on the log being directly editable. And realistically the log also needs to be the input format; it would not be reasonable to expect the admin to be able to edit a more formal log representation correctly (whether the log format is XML, protocol buffers, JSON, or something else). But in the case where the log format and the move input system match, just playing the game has taught the game admin the necessary skills.

This is a very nice feature for friendly games. It does have downsides though, more on that later in the section on the social implications.

There's also a potential as yet unimplemented feature of pre-programmed actions, that people frequently ask for. "I know exactly what I want to do next turn, why can't I just pre-enter my move". This would be a pretty interesting thing for speeding up games, but to my mind would not be conducive for good play. Circumstances change, often in ways you did not anticipate at all. The only way this could be even remotely usable would be if the language was extended to have some kind of conditional execution. And that's a can of worms I'm interested in opening, and I suspect also a bridge too far for 99% of my users.

It's worth noting that many of the above features are closely tied to a game with no randomness (or at most setup randomness) and no hidden information. As such their existence is something of an anti-feature, preventing other additions to the game.

For a non-hypothetical example, I'm currently thinking about how to implement the faction auction variant from the TM expansion. A full open auction in the beginning would be painfully slow. The most obvious, though still slightly imperfect, solution is a series of blind second price auctions. But this is not a good fit for the site's existing design. The problem is that the blind bid introduces momentary hidden information into the game, and it's possible for that information to leak through either the preview or admin modes. For example the admin could wait for everyone else to bid, peek into the log and see everyone else's bids, and then bid in such a way as to force the winner to pay the maximum amount.

UX

The most obvious UX consequence of using a command language is that it tends to be harder to learn. The following quote, said partly in jest, certainly contains a kernel of truth:

... has done a bang-up job providing a PBEM Terra Mystica experience that includes just enough extra layers of complexity via the interface and game administration tools to keep TM as confusing as ever, long after you master the actual game!

Non-natural languages are simply not a mode of human computer interaction that most people are comfortable with in this day and age. It actually continues to amaze me that I could get non-programmers to play using this implementation at all. Is it possible to evaluate how big a hurdle this has been for people? The best number I can come up with is that around 20% of the players who joined at least one game never finished even one game without dropping out. Note that these are players who have already jumped through hoops such as email validation during account registration. It's possible that there's some other issue beside the UI that's a problem for these players, but it does seem like the most likely candidate.

A smaller problem is that it essentially forces the introduction of a move preview. For those who haven't played the game, when entering moves you need to first enter the moves, then click 'preview', check that the results match what you want, and finally click 'save' to commit the moves. In a game that uses a direct manipulation paradigm, a preview could be skipped. But with a more obscure UI like here, it's absolutely essential since the move might not have had the intended effect. Whether it's doing the entirely wrong move, picking the wrong tile, building on the wrong location, etc. Even with a preview step somebody will request a rollback on average once or twice a game.

So why do I call this a problem? Because despite my best efforts, especially new players will frequently forget to 'save', leaving the game in a limbo state where they think they've done their move, until some other player gets impatient. (To mitigate this a little, the system will automatically do a 'preview' when using the GUI tools to generate the commands rather than type them. Unfortunately performance problems make it unfeasible to trigger continuous parsing + updates when typing).

A horrible mistake I made in the design of the language was the lack of (mandatory) turn delimiters. Originally my implementation treated each row as a complete turn. This caused more confusion than any other part of the command language. In the end I ended up writing a lot of very complicated code for automatically detecting the turn breaks in a command stream.

But that wasn't actually good enough, there are valid command streams where the splitting isn't unambiguous, e.g. the tunneling ability of dwarves, where transform E10. build E10. I had to make an arbitrary choice on that (basically the behavior now is greedy, as many commands as possible are stuffed into the same move). So I had to include the done command to allow players to disambiguate in the few cases where it's needed. This is still supremely confusing for people. All of this could have been avoided by taking this into account right at the start.

Finally, one very surprising outcome is that having a compact vocabulary for game actions makes it much easier to display a useful player-readable log of what happened in the game. The typical user-visible log is structured as natural language, and so verbose as to be hard to read especially when trying to piece together the flow of the game after the fact. It's easy to see why that design choice is made, but it's not necessary when all players are almost by definition going to know how to read a more compact representation.

Likewise this makes it really easy to display a concise summary of what has happened in the game since the player last looked at it (done both in the notification emails and the 'recent moves' tab of games).

Social issues

The unlimited admin access to games has a dark side. Admin malfeasance is rare but I do get about one complaint a month about it. Sometimes these are games where the admin will change their moves after others have already taken moves, rolling the game back by a huge amount, taking over entirely for another player for example forcibly passing them, applying different standards to allowing others to undo vs. doing it themselves, and so on.

This is the kind of drama that I really do not want to deal with, but the general solution is to just mark the game as unrated, and let the players sort out between themselves whether and how the game will continue. And it is a bit of a miracle that it hasn't yet become a more widespread problem, as one might expect to happen for the anonymity + internet combo. If it does ever become intolerable, the solution will almost certainly be to disable admin mode entirely for public games. The TM tournament has already shown that it's at least workable, even if people do occasionally get a little bit screwed by the 'no manual administration' policy.

One consequence of a command language is that everything needs to be named. The map needs to have a coordinate system, every component needs a identifier of some sort, and every interaction needs a short and snazzy name. Old school wargames will do this as a matter of course. Of course every hex has an id! Of course the cards are both numbered and uniquely titled! But not so much for eurogames.

The naming we ended up with on the site is far from optimal, and caused yet more drama due to non-online players feeling excluded from conversations. (If you want to know more, you can see an explanation for where the names came from, and why they won't change). That bit is unfortunate. But at least I actually find real value in having convenient shorthands available for everything, when discussing the game, whether when theorycrafting or conducting some tabletalk on IRC during a game.

Implementation issues

The obvious problem for a log replay system is performance. Replaying a full game, which is done for almost every operation, can take around 0.15 seconds in the current implementation, with no obvious low hanging fruit to fix. On the current traffic levels server load is not a problem, but I would start to get worried if usage increased by a factor of 10. As discussed above, there are features I'm unwilling to implement due to CPU load concerns. And it is actually causing real development pain for testing (see below).

It's hard to say exactly how much of the CPU overload is related to command parsing, a step that could be avoided with the use of a more structured log format. Some crude profiling suggests that the parsing takes only 5-10% of the runtime, certainly nowhere enough to warrant using a different format.

A rewrite in a language with higher performance implementations than Perl would almost certainly give a factor of 10 improvement on the actual game evaluation code, moving the bottlenecks to IO. But a full rewrite is not in the cards.

Another potential implementation worry is storage. The current DB size is about 250MB. Unlike CPU usage, this is a cost that accumulates over time. Out of that 250MB maybe 75% is used by the game logs. The logs, stored as a sequence of commands, are not a particularly efficient form of encoding the game data. Simple lossless compression could easily compress them by 80-90%. Luckily disk is cheap (this server still has 600GB free), so this should never become a real issue.

Another consequence of a log replay system is that any change in the game evaluation might break existing games. That change might be a bugfix for a place where the effect of a move was miscomputed, it might be extra validation to prevent illegal moves of some kind, cheating prevention, or something else entirely. This is not a theoretical possibility. Basically every single game evaluation change I make, there are already multiple affected games. No matter how elementary a rule is, somebody has already broken it.

Obviously in a stored state implementation changes like this don't matter. The current state is the current state no matter what. But in a log replay system you need to have some story on how to deal with retroactive changes. I can think of the following strategies:

Punt: Don't make any changes at all.
Ignore: Just make the change, and don't worry about games breaking or the results changing part way through.
Delete: Just delete any games that would be broken.
Fixups: Find all games where the old and new behavior differ, and change the appropriate logs in such a way that the results with the new log and version will be the same as the result with the original log and old version. This change could be manual or automated.
Versioning: Each game file carries a version number. When making a breaking change, keep both the original and new code paths, and choose one of the two based on the version number. Any newly created games use the new version number and get the fixes, existing games keep their original version number and the original behavior.
Positive options: Conditionalize the behavior on an option. Turn that option on for new games, as well as any existing games for which the new and old versions behave the same.
Negative options: Conditionalize the old behavior on an option. Turn that option on only for existing games where the results for old and new versions differ. Never turn the option on for newly created games.

During the lifespan of the site I've used most of these at one time or another. The 'ignore' strategy was appropriate a couple of times (for changes where I decided that the the new behavior was always acceptable, such as situations where a player had ended up overpaying for an action). The 'delete' strategy would be exceptional, the only situations where I used it were games that were aborted, and one case of a single game being completely unsalvageable due to bug abuse by a player. The 'fixup' strategy has the nice benefit that it avoids introducing a new code path, and was my default choice early on. But at this point it'd be an unacceptable amount of manual work, and it's not readily automatable. Especially with the relatively freeform input from the command language. My next default was 'positive options', but after about 3-4 of those I switched to 'negative options'. Positive options had a slightly more complicated rollout procedure, and also permanently clutter up all games, confusing people. ("What's this strict-darkling-sh option?").

None of these options are good, in this instance a log replay model does introduce some major costs either to the developer (who has to do extra work) or the users (who have some games screwed up or completely lost).

But it's not all bad! A log replay model makes testing much easier. First, it'd be very easy to write test cases since there is a very natural serialization format for games already, the command language. I don't actually write explicit tests for TM, but for example at work we need absurd amount of infrastructure for making it easy to write unit tests for TCP/IP packet handling. This kind of design gives the test cases for free. Likewise a Age of Steam implementation I was once doodling around with had lots of test cases, but even with the reasonably friendly format (protocol buffers) they were an absolute pain to write due to the boilerplate.

If I don't write unit tests, how do I test? Mostly by side by side testing; I have a small script that runs every single game in the database against both the new and the previous version. It munges the results a bit removing known harmless diffs, and then displays any changes from game to game. I can then look at those games, and decide whether it's indicating some kind of a problem with my change, an expected result of my change, or a problem of some sort in the game. It also acts as a great regression test that prevents failures from creeping in, and is the source of data for finding the games that would be broken by a game, so that one of the fixes discussed in the previous section can be applied.

This has been one of my favorite forms of testing for a long time, and works tremendously well in a case like Online TM where we have access to all games ever played. Thinking specifically of digital boardgames, it's also a model that wouldn't work well without a replayable log. The only problem is, as alluded to above, the CPU usage. Right now a full diffgame run takes about 90 minutes of CPU time on a rather beefy machine. Even with parallelization it's not a fast feedback cycle. (Makes me kind of miss being able to just casually run a sxs test on a thousand machines).

Conclusion

I'm afraid this ended up longer than intended, despite only covering one design decision. It's also a design decision that I feel is overall a win. You'll have to wait for the next post for the embarrassing technical missteps.

A brief history of Online Terra Mystica

jsnell@iki.fi — Thu, 27 Nov 2014 21:00:00 GMT

What's this Online Terra Mystica thing?

For the last couple of years my main hobby hacking project (over a thousand commits, and probably an order of magnitude more time spent on it than all other non-work projects combined) has been an asynchronous multiplayer web implementation of the brilliant board game Terra Mystica (Feuerland Spiele, 2012). At the moment it's roughly 2/3 Perl, 1/3 Javascript, and uses Postgres as the data storage.

It's been a fairly successful project for something that was originally intended as a one-off. The usage statistics at the end of November 2014 are:

Almost 6000 registered users
About 1200 monthly active users (as in playing at least one game; not passive use like looking at the statistics pages).
14000 moves executed on a normal weekday (10000 on weekends)
16500 games either ongoing or finished.
Bi-monthly online TM tournament run by Daniel Åkerlund with 400+ players.
1038 commits as of this writing.

This was not supposed to be a general use program. It was originally a one night hack to help keep track of a hand-moderated play-by-forum game of TM, which was obviously headed for failure due to the massive amount of errors people were making while describing their moves in natural language or when manually tracking their resources in the game.

From there the project snowballed, slowly gathering features including just about everything I ever marked in the TODO as being 'out of scope'. Since I often had only very limited amounts of time to work on this, and my expectation was always that the interest in the site would soon fizzle out, the project management method was to always get the maximum short-term bang for the buck.

A project whose direction is literally guided by 'what can I get done in the next two hours' is of course massively path dependent; the early decisions made with very little consideration had outsized influence on where the site ended up. Sometimes the expedient gambles on 'do the simplest possible thing' failed, and the results were just rubbish. At other times things ended up at a slightly odd local maximum. And in some rare cases the gamble turned out to produce wonderful and unexpected results.

Timeline

Future posts will discuss the actual lessons learned; what didn't work and what did work - both in the mechanics of programming and in the peculiarities of online boardgames. But in this one let's just have a look at the history of the site, how long it took for it to get features that one might consider absolutely necessary, and how amazingly bad user experience people are willing to put up with when it's the only way they can play their favorite game online.

Feel free to skip past the bulleted list if you get bored, it's still a bit long even if I include only changes I consider fairly major (indeed, a lot has to get filtered out given it's 1000+ commits).

2012

December - Early January: The smallest program that did anything useful related to a game. I'd enter moves into a text file and run the script to produce the final game state as JSON. This JSON was rendered to HTML + Canvas by some Javascript code that was half ripped off from an old project. There was some minimal rules checking and automation, and support for only 5 out of the 14 factions in the game. Users of the current site might want to see the old look.

2013

January: A rudimentary dynamic web site, implemented simply as a wrapper CGI script around the JSON generator script. After that a clumsy web-based editor was added for game files (a textarea that could be used to edit specific files in a git repository, no authentication except for each game having a random 160 bit identifier as part of the URL). This allowed other people to moderate their own games, as long as I created a game for them and sent the link with the secret embedded. Players would post / email a natural language description of their move to the moderator, who would then enter the moves into the admin tool using the correct syntax. Amazingly some 20 games were run using this insane system, while by all rights the project should have died there.

This version of the software had automation for resolving the effect most game events, but did very little validation to notice completely invalid moves.
February: Added an ability to easily rewind the game state back to any time in history, to help with post-game strategy analysis. Also added a way for players to enter their own moves (a textarea in the main game view, a preview button and a save button, and some verification to make sure they could only enter their own moves). Again there was no real authentication here, just links with an embedded faction token derived from the per-game secret key.
March: The hackiest email integration in the world: Store the email addresses players in the same text file with the commands. After a player has entered a move, the software would create a mailto: link with prefilled subject, content and receivers (the other players). The player would clicks on the mailto: link, the email loads up in their mailer (even GMail), and they'd press send.

Compute and display a VP projection on the last round assuming no further moves, to give players some idea of who is really winning.
April: I continued to resist adding any user management or authentication. But my friend Gareth wanted a better way to manage his ongoing games than a spreadsheet, and wrote a small App Engine site into which players entered their secret game URLs. His site then used my site's API to figure out which games the player needed to act in. And it went even a bit further, by embedding the move entry UI into the same app.

After a few weeks of using Gareth's site, I had to admit that he was totally right about this being required functionality. So I finally added a DB to the project for storing user accounts and game metadata, and a 'your games' list on the front page after login. It's also only at this point in the lifetime of the site where I added a UI for people to make new games. Until then every game was created by somebody asking for a new game via email.

Finally, this month also saw the addition of a statistics page on how often each faction was winning (since balance was a hot topic on the BGG forums of Terra Mystica right from the start), and soon after a list of achieved high scores for each faction and player count.
May: This month mostly introduced all kinds of stricter validation, as the reduced barrier to entry for playing was causing significantly more illegal moves to be entered (early on players were enthusiasts of the game and thus had good knowledge of the rules; at this point people started to learn the game through the site, which was quite scary).

The main new feature of the month was the 'planner', an alternate text entry box which could be used to enter commands arbitrarily far into the future, and check that the moves are valid and what kind of effect they have. This is useful for example for checking that you have sufficient resources for making certain moves without manual computation. Another use is leaving 'notes to self', so that the player doesn't need to re-evaluate the board for every single move. (Some people were suddenly playing tens of games at a time, so this was a real problem).
June-August: This time period saw only minor fixes and improvements from the user's point of view. There was a bit of infrastructure work behind the scenes, such as moving the actual game moves into the database, though they still remained just plaintext.
October: The mini expansion for TM was released at the Spiel fair in Essen. I implemented the new features the very next morning in lobby of my hotel at Essen, with a ChromeBook, a ssh connection to the production server, and and the world's worst WiFi. After some reflection I decided not to make the change visible to the public before getting back home and a more reliable work environment :-)
November: I finally made the site automatically send email notifications, rather than require players to jump through the fragile mailto: hoops to let other players know whose turn it is. Replacement of the mailto-style notification of moves also required the addition of an in-site chat feature for communication.
December: Another consequence of the real email support from the previous month was that players no longer needed to expose their email addresses to other players. This finally made it possible to allow players to create 'public games' that anyone can join, rather than only play people with whom they've done some kind of an out-of band email address exchange. (At this point 1500+ games had been started, amazing how far such a kludgy system could go).

At the time 25-30% of moves were being entered from smartphones or tablets. But the move entry interface was typing commands like 'convert 2pw to 2c. upgrade d3 to tp' into a text box. What's wrong with this picture? :-) In the month we finally got a slightly friendlier UI, though the textual command representation still remained the canonical one.

The site finally got a ranking system: a multi-iteration version of the ELO algorithm, which computed not only player strengths but also faction strengths, and credited good results with the weaker factions more than good results with the strong ones.

Finally, in very late December I went on a big refactoring spree to move the game from CGI scripts to a more persistent application server (FCGI with Plack and CGI::PSGI, but no framework). Eradicating all global data and all modification of literal data structures was way too much work, those were not corners worth cutting in the first place.

The new UI went live a year from starting the project (almost exactly; from December 22nd 2012 to December 21st 2013), and is the point where I'd consider the site to be actually usable by mere mortals.

2014

February: Support for variant maps, for testing parts of the upcoming Terra Mystica expansion for the designers. I also added a map editor that could import map definitions from Lode's TM AI, which the design team had been using for the map. The online playtest team proceeded to play 100 games with different map versions before the expansion finally went to print.
April: A bunch of work on the expansion, which was still being kept under wraps. So the support for the new final scoring types and four of the new factions was not visible to most users at this time.

The main user-visible change was automatically dropping players from games after a week of inactivity, to support the inaugural season of the online Terra Mystica tournament. People's irritation about others playing slowly had been constant ever since the addition of public games (95% of my games are private with a few separate groups of friends, so I'm pretty isolated from this myself). Unfortunately this change appears did not appear to help enough.

This month also saw the addition of individual profile pages, showing all kinds of statistics for each player (games started, finished, performance with given factions, performance and play counts against specific opponents, etc).
September:The next attempt at reducing the anguish caused by slow players was to allow setting shorter move timers than the default one week (from 12 hours to 14 days). Lots of people started 12 hour deadline games, and moved on to complaining about so many people dropping out. Sometimes you just can't win.
October:Public support for the two new expansion maps, as well as the new final scoring types.
November:Public support for all six new factions from the expansion, as well as the variable turn order variant.

I find it interesting that it really did basically take a year of real time (and maybe 2 months of hacking time) before the implementation was in a shape where I would've thought about publishing it. And there's no way I'd put that amount of time into a project like this up front. Usually these projects are active for a couple of weekends before getting abandoned; fun parts are done but all the hard work of making it really usable remains.

In this case people were eager to use even the incredibly crude early versions, so I got over that hump very quickly. And at that point every incremental improvement to the site was affecting tens, hundreds, or thousands of people. This is of course always more motivating than working on polishing the perfect piece of software that nobody is using.

There were many architectural and design decisions done along the way that I ended up deeply regretting, and which cost me lots of time later on. But without all those early shortcuts there would've been no implementation at all. Easily the best example of Worse is Better that I've been personally involved with.

Feed moving

jsnell@iki.fi — Wed, 04 Jan 2006 05:00:00 GMT

I've moved my blog away from the University of Helsinki Department of Computer Science servers, where it's been living for almost two years. In anticipation of this moment I originally made all the links to my rss feeds through as forwarding service. When the blog moves, just flip the redirector to point elsewhere, and the users won't notice a thing!

At least that was the plan.

Apparently a lot of people still managed to subscribe with the target URL of the forwarder (http://www.cs.helsinki.fi/u/jesnellm/blog/rss-...), instead of with the forwarder URL (/blog/rss-...), and thus would still be seeing the old feed after the move.

So now I've just set up a HTTP 301 (permanent) redirect on the old location. Smart RSS aggregators are supposed to update the feed URL when seeing a permanent redirect, but judging from the access logs few do this in practice. Instead a 301 is treated the same as a 302 (temporary) redirect.

Which brings me to the actual point: If you want to keep on subscribing to this feed indefinitely, please check that you aren't using the cs.helsinki.fi URL. I'm hoping to graduate this year, which might also imply losing the 301 from the old location to the new one.

While moving servers, I also took the opportunity to redo the blog as a dynamic application (using Araneida) instead of generating static pages. I'll see whether I can procrastinate myself into adding comment support in the future.

Golf - Deroter

jsnell@iki.fi — Sat, 23 Jul 2005 10:00:00 GMT

Long time since the last golf. Inspired by the recent announcement of a Perl Golf book I took part in a Polish golf that was announced on the mailing list.

Given a input string that has been "encrypted" with ROT-n on STDIN and a dictionary of words (sequences of letters A-Za-z, not of \w) in @ARGV the program needs to output to STDOUT the original plaintext. (Formal rules).

My best solution was 62 characters, but I figured out about an hour before the golf ended that it was actually broken, and didn't have time to figure out anything better than the 65.44 below, which is currently good for a second place. The apparent winning solution of 63 doesn't seem to work either, for unrelated reasons. So the explanation might be for the winning entry, or it might not.

#!perl -p0

You know the drill. -p handles reading the input and printing the output. Use -0 to read the input in one go, instead of a line at a time.

INIT{%a=map{pop,1}@ARGV}

In the INIT block, pop all command line parameters to make -p read from STDIN. Use the removed arguments as keys in a hash table for detecting dictionary words. Using the symbol table with something like $$_=1while$_=pop would save a few characters, but that's incorrect since $ARGV is automatically set to '-' on entering the main loop.

$a{$&}||y/B-ZA-Gb-za/A-z/while/\pL+/g

At the start of the main body $_ contains the whole ROT-n text.

On the first iteration /\pL+/g will match the first word (letters only; \pL is essentially [a-zA-Z]). //g works differently in scalar than in list context: it will only match once per call, but the next call will start at the location in the string where the last match ended. If a match was found it returns true, otherwise false.

In the body of the while we first check if the word we matched is in the dictionary. If it isn't (i.e. $a{$&} is untrue) $_ obviously isn't plaintext yet, so we rotate it by one step with y///. This contains the only tricky bits in the program:

Changing $_ causes the scalar //g to be reset, and start matching from the start of the program.
Doing the rotation backwards (A -> Z, B -> A, ..., Z -> Y) instead of the more intuitive direction (A -> B, B -> C, ... Z -> A) allows writing the transliteration in a way that saves one character.

There are six characters ([\+]^_`) between Z and a. By adding six extra characters into the right place on the left side of the transliteration operation (with -G) we can use the range A-z on the right side, instead of specifying separate ranges for upper- and lowercase letters. Compare:

y/A-Za-z/B-ZAb-za/
y/B-ZA-Gb-za/A-z/

FWIW, the 65.48 by Piotr Fusik by far the coolest solution. Wish I'd thought of that...

Golf - Rush Hour

jsnell@iki.fi — Thu, 01 Jul 2004 00:00:00 GMT

The recent godzillagolf titled Rush Hour was the first golf in a while where my solution contained anything worth explaining. Here's all 157 characters of the solution:

#!perl -n0
sub
R{$b<0?reverse:$_}sub
M{/
/?s^\pL^$b=$#A**pos;push@_,"$& $b
";$c=8*($<
Z);s/$&/ /,s/(($&)\C{$c}) /$1$2/<++${$_=R}or&M
for~~R;pop^ge:exit
print@_}M

The code uses a depth-first search, which can be roughly divided into the following steps (the actual code doesn't do things quite in this order):

If current board has already been visited, backtrack to step 4.
Mark current board as visited
If a car is in the target space, print the moves that have been accumulated and quit.
Move one of the cars one step in either direction
- If no cars can be moved, backtrack
- If backtracking to here, try another car/direction
Go to step 1.

There are several subproblems that need to be solved to implement the algorithm:

Detect whether the win condition has been reached.
Accumulate the moves. (Preferably without using too much memory; I have a 149 character solution that uses 200MB, which isn't really justifyable for this problem).
Iterate over all valid moves (car/direction pairs) for a board.
Given a board and a move, generate another board.
Ensure that the backtracking works.
Detect whether the board has already been visited

The board is of course stored in the original format as a string. There's no room for any fancy datastructures... Given that, here are my solutions to the subproblems:

Just check whether the board contains a space followed by a newline:
```
/ \n/ ? ... : ...  # If the regexp fails, win)
```

Keep the moves stored as strings in @_ in the correct order:

push@_,"$& $b\n"; # $& is the current car, $b is either -1 or 1
...   # execute the rest of the algorithm. This quits on success...
pop;  # ... so reaching this line means that we're backtracking,
      # and need to remove the move

Given a board, iterate over all valid car-characters in the string (I used \pL here instead of the obvious \w for reasons that are still unclear to me). For each character, generate a value $b as either 1 or -1, so that it's guaranteed that for any board both values are generated at least once for each car. Since each line is 9 characters long and there are at least 2 characters in each car, each car must have at least one character in an even and one in an odd position in the string. Hence (-1)**pos generates a proper value.
```
s^\pL^$b=$#A**pos; ... ^ge
```
$#A is just a shorter way of writing (-1). Unfortunately the operator precedence of unary - is smaller than that of **.

First let's solve the problem only for positive values of $b (i.e. down or up).

$c=8*($< Z);  # $c = 8 if car moves up/down, 0 otherwise
s/$&/ /;        # Remove the first character of the car
s/(($&)\C{$c}) /$1$2/
# Find last character of car that's followed by a space exactly $c
# characters from it, and substitute the space with the character.
# For example "bbcc." => "bb.cc". If this substitution fails,
# the move was impossible and we should backtrack.

To handle negative values of $b, just conditionally reverse $_ before and after it's modified (nobody else did this in the golf, which I found suprising):

sub R{$b<0?reverse:$_}
$_=R;...;$_=R

The backtracking can be implemented just by wrapping the code inside a recursive subroutine and restoring the original state if the recursive call returns. There are three interesting bits of state:
- $_. Saved by binding $_ again with for:
```
... for"$_"  # Can't use ... for$_, since that just
               # aliases the current $_ to the new $_
```
  Since $_ needs to be conditionally reversed in d, we can just use the return value of R instead.
```
... for~~R   # ~~ needed to give scalar context to the reverse
```
- The moves that haven't been tried for this board yet. Since these are generated from the substitution in subsolution 3, nothing special needs to be done.
- The accumulated moves. This was handled correctly in subsolution 2
Mark board visited by incrementing a symbolic reference using $_.
```
++$$_
```
Since $_ needs to be conditionally reversed again, the symbolic reference can be made on the value of the assignment instead:
```
++${$_=R}
```
The other part of this subproblem is to not recurse if the board has already been visited. This can be done by comparing the return value of the increment to the final substitution in subsolution 4:
```
s/...//<++${$_=R}or...
```

Mash these ingredients together, and add an exit print@_ to actually do something with the result, and you get the solution shown above.

Golf - Matrix

jsnell@iki.fi — Tue, 13 Apr 2004 00:00:00 GMT

The Matrix golf had a rather thin field (hopefully only temporary), but some really cool code (especially in the post-mortem).

The problem statement was short enough to be quoted here in full:

Let A be an N*N matrix of zeros and ones. A submatrix S of A is any group of contiguous entries that forms a square or a rectangle. Write a program that determines the number of elements of the largest submatrix of ones in A . Largest here is measured by area.

Before going into the details, a brief example of how the algorithm I used works. Assume the following matrix:

The longest string of 1s is the 111 on line 3, so that's the largest submatrix of one line (with an area of 1*3=3). Then transform the matrix by doing a stringwise and on each line and the line that follows it. The last line will be chopped off:

Obviously the only way to get a 1 on the transformed matrix is to have one on the corresponding position in two successive lines in the untransformed matrix. So the string 11 (found on both lines 2 and 3) corresponds to an area of 2*2=4 in the original matrix. Repeat the transform:

00010
00110
00010

Now any 1 is going to be the result of 3 1s on consecutive lines, so 11 on line two means there was a submatrix of area 2*3=6 on the original matrix. Repeating the whole process two more times would result in finding an area of 4 and one of area 5. The answer for this matrix would therefore be 6.

My solution (59 characters):

#!perl -lp0
s/1*/$B[$?*length$&]=$&/ge,/
/,$_&=$'
while++$?;$_=$#B

As usual, we slurp the whole input into $_ with -p0 and take care of the trailing newline with -l. In addition to $_, a couple of other variables contain some interesting state. @B is used for keeping track of the largest area that's been found (an old golf trick; we're only interested in the size of the array, not the values stored in it). $? holds the current iteration (i.e. the multiplier for the area calculation). $? is used since it can only contain an unsigned short (0-65535), and therefore repeatedly incrementing it in the condition of the while results in the variable overflowing to 0 after 65535 iterations. (Another variable with a similar behaviour is $^C, which holds signed chars. I used $? instead since at some point my program couldn't handle negative multipliers)

As mentioned before, the program contains a while-loop whose condition is just incrementing $?. The body of the while implements most of the algorithm. Some code is executed for each string of 1s with s/1*/.../ge. The code in question is $B[$?*length$&]=$&, which just calculates the area of the submatrix that the string of 1s represents (by multiplying $? and the length of the matched substring, i.e. $&), and stores something in that index of @B. In this case, the value being stored is $& since (despite using s///) we don't actually want to modify $_ yet. This takes care of finding the largest area.

To implement the transformation described earlier, a /\n/ is used to find the first newline in $_. After this a stringwise and of $_ and $' will have done the tranformation (including chopping off the last line). Once the loop ends, we just assign $#B (the largest index of @B) to $_, which is then printed out thanks to the -p command line argument.

This was a very cool golf. My only regret is missing a completely obvious optimization of replacing the s/1*//ge with a suitable crafted map, which would've saved two strokes. Well, not completely obvious since I only realized that it would've been possible when writing this post. Perhaps I should start writing these things earlier... ;-)

Golf - Subproduct

jsnell@iki.fi — Sun, 21 Mar 2004 00:00:00 GMT

For some reason I usually don't seem to have time to take part in golfs that I've designed. Almost happened with the badly named (couldn't come up with anything better) Subproduct too, but in the end I decided that I didn't really need to write that seminar report yet...

The problem was simple. Given a string of digits (maximum length 20) and a maximum substring length N (maximum of 9), find the largest product of the digits in a substring of 1..N characters. (For example for the string 0120340 and a N of 5 the correct answer is 3*4=12). The most complex parts about the problem are handling zeros correctly and keeping track of the maximum value encountered (the usual golf idiom of using the length of an array for this doesn't work, since the maximum value of 9**9 would require an array that's too large).

Here's the code (68 characters):

#!perl
$_=shift;s/./$^=1;($^*=chop)<$\or$\="$^
"for($`.$&)x"@ARGV"/ge;print

First we get the first command line parameter into $_with shift, and loop over it using s///. The answer will be saved into $\ so that we can use just a print without any arguments to print it. Of course print without arguments will print $_too, so we need to empty $_ somehow. This is the reason for using s/.//, while the shorter s/// would otherwise also suffice.

For each position in the input string we'll first initialize $^ to one. After that we'll loop N times through a loop, where $_ has been initialized to "$`$&"(that is, all characters up to and including the one that's currently being processed by s///). In the loop, we'll chop off digits from the end of the newly constructed $_ and multiply $^ with them. If $^ is larger then the current value of $\, set $\ to "$^\n".

There are a lot of variations on this theme that are equally long. I ended up submitting one of the more obfuscated ones, only to regret it now :-)

Golf - Card Trick 2

jsnell@iki.fi — Wed, 03 Mar 2004 00:00:00 GMT

After missing one minigolf, I had some time to take part in Card Trick 2.

The mission was to determine the result of the 'trick' outlined below, when getting as input the initial layout of cards and the actions of the 'audience'.

In my hand I have 21 cards that I deal out face up, one each to 3 spread piles, until there are 3 rows of 7. You silently chose your card and inform me of which 'pile' your card is in (1, 2, or 3). I then pick up each pile making sure to put the pile with your card between the two other piles. I deal them out as before, and again you tell me which pile your card is in. We repeat the process a third time, and when I again pick up the piles, placing the pile with your card in the middle, your card will invariably be the center card (11 of 21 in this case).

By the time I'd even read the rules, there was already a mass of people with a solution of 40-41 (and Ton Hospel at 38, but we all know that he isn't human). Given the length of those solutions, it's obvious that the solution is going to involve some cute mathematical formula, instead of directly manipulating the cards to execute the trick.

The way I thought of the formula was this: Given an index I into the set of cards (for example with the 21 cards below, 0-20) and the pick P (1-3) we need a formula for determining from I and P the index which would get translated to I.

Let's determine this by hand for for the interesting elements (we're only interested in cards that are rearranged into the middle third):

    7  8  9 10 11 12 13
 1  0  3  6  9 12 15 18
 2  1  4  7 10 13 16 19
 3  2  5  8 11 14 17 20

Obviously our formula looks like I' = P + 3I - 22. To generalize this we just note that 22 is the number of cards + 1 (which happens to be @F-2). Now, to solve the problem we just need to remember that the card that we're interested in ends up in the middlemost position (i.e. for 21 cards in index 10, @F/2-2), and we can just repeat the formula three times to find out the original position:

  I_0    = @F/2-2
  I_1    = P_3 + 3*I_0 - (@F-2)
         = P_3 + 3*(@F/2-2) - @F + 2
         = P_3 + @F/2 - 4
  I_2    = P_2 + 3*I_1 - (@F-2)
         = P_2 + 3*(P_3 + @F/2 - 4) - @F + 2
         = P_2 + 3*P_3 + @F/2 - 10
  I_3    = P_1 + 3*I_2 - (@F-2)
         = P_1 + 3*P_2 + 9*P_3 + @F/2 - 28

So that's the theory. In practice few Perl tricks are needed for this problem. The cards can be accessed from @F by turning on autosplitting with the -a switch. The picks could also be accessed from @F, but it turns out that it's easier to access them using regexps. By crafting a suitable regular expression, we can get the picks into the special regep variables ($&, $', $1, etc). And finally, by using a repeated substitution, we can use the return value of operation to count the amount of cards in the input (to save one character when compared to using @F/2. My final solution is 39 characters (and a shared second place):

#!perl -lpa
$_=$F[s/ ..(.)//g-27+9*$'+3*$1+$&]