After I finish releasing the RAF versions of the Echelon Reference Series, I’ll be rebuilding my workflow for data capture. It will be simpler and more abstract, and more powerful. I thought it might be interesting to show how this has evolved.
Origin of the ERS
The ERS started as a set of research documents for Echelon. There have been several iterations.
Originally I just created a file for each topic — feats, spells, rage powers, etc. — and dumped all the relevant bits into the file. This worked for a while, but I started running into places I wanted to split the data up or organize it in different ways. The feat document could be split into item creation feats, metamagic feats, combat feats, style feats, and… the last two could be applied to the same feat.
And the prerequisites. The complexity of the prerequisites started to overwhelm me, and there wasn’t a lot I could do about it. First design scrapped, thankfully before I invested too much in it.
I started to use styles to mark data types, with a style for each data type of interest. I didn’t limit it to ‘feats’ and ‘spells’, either: rogue talents, advanced rogue talents, domains, all had their own styles (that mostly looked like Heading 3… I didn’t start changing the style presentation until later). Using heading styles for game elements made it easy to navigate my Word file because they’d show up in the navigation pane, but making them look the same meant it was easy to assign the wrong one.
I was manually saving the files as Word and ‘filtered HTML’ formats so I could do a series of transformations to parse and extract information. Everything was done in a single directory… don’t do that. Hundreds of source files, each combined with about eight intermediate files before hitting the final form made for a huge directory full of stuff I don’t care about, with important stuff inside it.
More intermediate files than that, even. After parsing the information I would spray out the individual items into their own files — each feat, each spell, each rage power, etc. — that would be picked up by a script for inclusion in output. I started building PDFs via LaTeX, but it wasn’t very manageable. It was also very easy to accidentally delete something.
… as I actually did. Got a new computer, was working on it and decided to blow away the ‘copy I’d made’, forgetting I hadn’t actually copied the files but created a shortcut to the folder to ensure I kept both versions in step.
Much to my surprise, after a moment of shock I wasn’t even particularly upset. No swearing or tears involved, even, just a deep breath. I was running into severe limitations and was ready to move on already.
This version saw the release of ERS: Barbarians, ERS: Clerics, and ERS: Sorcerers. This is where I started to abstract the data types. ‘Class feature’ had been around for a while, but I replaced all the individual feature types (rogue talent, etc.) with a more abstract ‘class subfeature’ type. This reduced the mental space needed to keep track of things, and simplified parsing later quite a bit.
Rogue talents and rage powers really parse about the same way, so why differentiate in code? I know from the parent class-feature what this is (class-subfeature with class-feature of ‘Rage Power’ means I’m looking at a rage power, right?) so I could mark it and move on. Lots of heavy encoding in the object markers, mostly for automation reasons.
Runeforger (Su) [1 Forgemaster’s Blessing; 2; 4; 6; 8; 10; 12; 14; 16; 18; 20] <channel energy>
A forgemaster may inscribe mystical runes upon a suit of armor, shield, or weapon as full-round action, using this ability a number of times per day equal to 3 + her Intelligence modifier. These runes last 1 round per cleric level, but inscribing the same rune twice on an item increases this duration to 1 minute per level, three times to 10 minutes per level, and four times to 1 hour per level. Erase affects runes as magical writing. A forgemaster learns forgemaster’s blessing at 1st level and may learn one additional rune at 2nd level and every 2 levels thereafter. Only one type of rune marked with an asterisk (*) may be placed on an item at any given time.
Using the runeforger class feature of the forgemaster cleric archetype as an example, The heading line has:
- class feature name,
- class feature type (Su, supernatural ability),
- [1…20] showing the levels it is applied (first level gives ‘Runeforger (Forgemaster’s Blessing)’, then every even level up to 20th you get another runeforger choice), included mostly so I could automatically create the level table correctly (or at least automatically),
- <replaces class feature> indicating that runeforger gets this instead of the ability to channel energy. <<double angle brackets>> indicates that this new feature only modifies an existing feature.
This actually worked, more or less. I could parse and render the items, and even eventually combine them (applying archetype to base class) to get the archetype class, complete with level table. I had almost no control over it, though.
Parsing, Prerequisites, and Pictures
This was also the version where I finally had the ability to extract and automate prerequisite links. This gave me the ability to draw pictures showing me the prerequisite relationships between game elements. The first pass would be machine drawn using GraphViz (software that takes a list of nodes and edges and draws the diagram — linked page includes sample diagram for the Improved Sunder feat), which was helpful for understanding but not very pretty when I tried to include them in the PDF.
I would then redraw the diagrams using PGF/TikZ, a LaTeX package that, well, draws diagrams. I’d write down where to put each node and how to draw the edges, then the image would be rendered and added to the output PDF. This let me put diagrams with hyperlinks in the documents.
For example, the diagram to the right shows the relationship between Animal Companion (top, second column over) and a bunch of feats (pale brown). In many cases Animal Companion is not sufficient, and the feat has other prerequisites (class features or subfeatures, mostly), and in others Animal Companion is one of several options to meet a prerequisite (one of animal companion or familiar or mount or divine bond… or some subset of these).
The other thing the prerequisite parsing and linking allowed was the automated discovery of ‘class-relevant’ feats. I could take the list of class features for the base class and archetypes, and look for feats that had those class features (or subfeatures) as prerequisites. I could then pull a copy of those feats into the PDF. Some of the choices ended up looking nonsensical because they required features from other classes as well, but since the barbarian’s uncanny dodge did meet the prerequisite I kept it.
This also is where I started working with an offsite Subversion repository. I had my formatting software on a server offsite and SVN allowed me to send minimal file changes from my workstation to the server, while making it almost impossible for me to accidentally blow the whole thing away again. I’m now creeping up on revision 2000.
In Second Cut I was manually saving the files as DOCX and Filtered HTML as I went. This was troublesome and annoying, and made it frightfully easy to get out of step — a couple times I ended up making the same change more than once because I’d saved as Filtered HTML and forgot to save it again as DOCX.
I wrote a program that would go through the ‘docx directory’ and convert all the files to Filtered HTML. Every time it was run. Very time-consuming and wasteful, but after learning of the ‘%’ file designator I switched to making the program a filter (single-input, single-output) and adding a make rule so only the files that were out of date needed to be replaced.
The third cut did a lot, but it didn’t give very good control over the output. There was no facility for being selective: the druid got access to some cleric domains, so if I included domains in the druid book I had to include all the domains. I could add a note to the front of the chapter that said “druids get only these ones, ignore the rest”, but after including subdomains that amounted to about a hundred pages when I needed only a couple dozen (which is why I didn’t release the druids book under the ‘third cut’). It also gave very little control over spacing the like, the small tweaks that let me fix unfortunate object placement. That’s why this version sees some places where a feat name appears at the bottom of a page and the description on the next.
The lack of control over layout was getting to me, so I took steps to clean the data up some more and regain more control over the layout. This led to the release of the ‘RAF’ (‘Rough And Fast’) releases that grabbed all the information, but didn’t add the diagrams or useful redundancy (and have a ticket price of 50% less than the final version will). I no longer capture certain information (such as the detailed level progression of the class feature within the class) and reworked the entire linking process so it could work on individual files as it parsed them, and was more accurate.
This cost me some of my automated data discovery, though. I no longer have the tools in place (with the current data set) to generate the initial version of each of the books. It also does not handle duplicate items well, something that really showed up (to me) with Draconic Bloodlines, when it could not differentiate between the ‘Claws’ bloodline powers. It seems a minor thing, but it bothered me.
So, while I did release ERS: Rogues, ERS: Fighters, ERS: Monks, and ERS: Rangers in RAF (except Monks, which is WIP — Work In Progress, the next stage), and will release the other ERS class books and the ERS spell books (sample ERS: Elemental Wizard Spells, PWYW), I will be revising the entire workflow and data capture process to make it easier to move to the next step.
And I’ll describe that in my next post.