DFHack mission, strategic objectives, and roadmap

[myk002]

DFHack project overview and mission statement

DFHack is an add-on for Dwarf Fortress that enables mods and tools to significantly extend the game. The default DFHack distribution contains a wide variety of these tools, including bugfixes, interface improvements, automation agents, design blueprints, modding building blocks, and more. Third-party tools (e.g. mods downloaded from Steam Workshop or the forums) can also seamlessly integrate with the DFHack framework and extend the game far beyond what can be done by just modding the raws.

DFHack’s mission is to provide tools and interfaces for players and modders to:

• expand the bounds of what is possible in Dwarf Fortress
• reduce the impact of game bugs
• give the player more agency and control over the game
• provide alternatives to toilsome or frustrating aspects of gameplay
• make the game more fun

Objectives

Improve usability and discoverability of DFHack tools

Why this is important: This is the top player-requested improvement to DFHack. DFHack users are frustrated with not being able to find relevant DFHack tools or use them effectively.

Example projects:

• Provide an in-game alternative to init scripts (control panel, persistent tool config)
• Provide useful presets for tools with complex configuration
• Make screen-relevant operations visible and accessible via discreet widgets (e.g. orders import on the manager orders screen)
• Make hidden keybindings more visible with interactive on-screen widgets
• Provide GUI interfaces for tools with complex command interfaces (e.g. tiletypes)

Have 90% of players upgraded to a stable -r1 release within 1 week of a DF version bump

Why this is important: Steam will update players to the newest DF version as soon as it is available (unless they specifically disable updates). Those players will be unable to use DFHack until DFHack makes a corresponding release and the players get that updated DFHack release installed. We can expect more frequent DF releases in the near future. DFHack will not be useful to anyone unless it can keep up. Mods won't want to use DFHack to extend their capabilities if they can't depend on DFHack's quick availability.

Example projects:

• Create test plan to validate tool functionality; start manual, but build supporting tools and automate the test plan over time
• Negotiate with Today/Kitfox to reduce the burden of reverse engineering (e.g. access to (scrubbed) header files, stable action and hook APIs, early access to representative binaries, etc.)
• Identity core features so a stable -r1 can be released quickly with basic functionality; will also need a mechanism for packaging a release that omits or hides broken/untested functionality.
• Require PRs to include functional regression tests (perhaps lift our usual requirement that tests should undo any changed game state to reduce test design burden)
• Isolate user config in dfhack-config dir so players know exactly what to delete and what to keep when doing a manual update (get rid of non-user-owned files in that dir, move more user owned files into that dir)
• Negotiate with kitfox to enable distribution of DFHack via steam so updates are automatic for most players

Be responsive to community needs

Why this is important: DFHack tools will be more useful if players can get their questions about them answered. An engaged and knowledgeable community will help others use DFHack effectively, and will help guide DFHack development. We will also have more negotiating power with Bay 12/Kitfox if they see that the community views DFHack positively.

Example projects:

• Answer DFHack-related questions and provide support on the forums, Discord, and Reddit
• Regularly poll the community for feedback on DFHack features and direction
• Regularly post progress on DFHack improvements
• Prioritize fixing reported bugs
• Steadily implement feature requests
• Review contribution PRs within one week of submission

Roadmap

See: https://github.com/orgs/DFHack/projects/23/views/1

[lethosor]

Require PRs to include functional regression tests (perhaps lift idempotency requirement to reduce test design burden)

What are you referring to by "idempotency requirement"?

Have 90% of userbase upgraded to a stable -r1 release within 1 week of a DF version bump

This is an ambitious goal. I think we'll have a hard time, at least in the near future with the resources we have. In the past, a month to three months has been more realistic, but with some dedicated effort we may be able to accelerate it. Some of it is dependent just on testing plans and resources, but some is heavily dependent on how much DF changes internally between versions, and that isn't something we can easily plan for. There's a decent chance, for instance, that not a lot of structures change in the next version, relative to previous major releases, but we can't know until it's in our hands.

"a -beta1 release" might be more realistic, but even then a week is probably a stretch. We've relied on public betas pretty heavily for ironing out lingering issues in the past.

Unsure if we have a way to measure "90%" unless Steam workshop provides that. We could look at our own download counts, I suppose.

Negotiate with Today/Kitfox to reduce the burden of reverse engineering (e.g. access to header files, stable action and hook APIs, early access to representative binaries, etc.)

Headers are probably out of the question, or at least would require significant dev time from Toady to make it possible. (It would include a lot of code and IP that Toady doesn't want released.)

We have asked him to expose a couple hooks in the past, which he has agreed to, e.g. SDL_NumJoysticks() and making certain methods virtual. The important part is ensuring minimal effort on his part and minimal risk to DF.

[myk002]

"idempotency requirement"

I'll reword that. I just meant that our current unit tests go to some lengths to ensure that the game state after the test is about equal to the state before the test, such that a dev could run tests during an active game and not worry about hosing that game. Widespread regression tests for game behavior is likely to be a one-way function unless we take great pains to restore state afterwards.

Have 90% of userbase upgraded to a stable -r1 release within 1 week of a DF version bump

I expect this to take quite a while to achieve (O(year)?), but I do think it's a particularly important goal for us. I don't want us to get into a cycle of toilsome, error-prone manual testing that can get stun locked by frequent bugfix DF releases. I want us to keep this goal in mind as we go through the next several releases so we can identify (and mitigate) the long tail tasks.

The important part is ensuring minimal effort on his part and minimal risk to DF.

I agree here. My hope is that we can find resources at Kitfox to help us do whatever we need to do. If there are specific requests we can make of Today, especially for relatively small code changes that will make our lives significantly easier, we can combine them into a batch that he might be able to address in <1d of work. Or maybe we could get him to do a technical AMA video chat that could help us solve problems that we're having trouble with.

[lethosor]

Yeah, integration tests are difficult to reset afterwards. We already have some ability to restart DF on failed (crashed) tests; should be possible to build in support for tests that require restarting DF even when they succeed as well.

Toady has been receiptive to small-scale answers to questions in the past (e.g. "how does this new structure work (and how do we stop corrupting the game)?" in #741 (comment), or "what are all of the flags in this struct"? in a FotF - that one surprised me a bit).

[oorzkws]

I want to comment that Factorio includes .pdbs with the Steam version, perhaps we could convince Toady/Kitfox to do the same so we have some basic info generated about the structures and changes?

[ab9rf]

while we would love for toady to provide pdbs, it has been our consensus that it's extremely unlikely that he would do so since they would reveal far more about the internals of his code than we believe he wishes to reveal.