Comment on incremental builds

Principles:

• Retain as much of the formatted output as possible from build to build.
• Maintain an individual build stream for each topic from souce to output. Where merge is necessary as during book processing, defer merge as late as possible.
• Save every stage of the build stream as an intermediate file so that the build can rebuild as late in the build stream as possible.
• Separate the aspects of building a topic that are specific to each map. Also separate the aspect of building a topic that is independent of any map.
• Defer conditional processing as late as possible to minimize rework prior to filtering.
• Use timestamps and deltas to determine when to rebuild.

While a full rebuild with these principles will take longer, an incremental rebuild should take much less time. For instance, moving conditional processing as late as possible means that we might process files that don't need to be processed for a specific build target. If we are building multiple targets, however, there is a considerable savings to building the file once.

Build steps and intermediate files:

Guided by those principles, we might end up with something like the following:

sourceTopic2contentDependency

input: 1 sourceTopic and every topic from which it references content
action: captures the content dependencies
output: one contentDependency file per topic referenced by the source topic
rebuilt: when the sourceTopic or the referenced topic is newer than the contentDependency file

sourceTopic2topicDependency

input: 1 sourceTopic and every topic from which it references the title or shortdesc
action: captures the topic dependencies
output: one topicDependency file per topic referenced by the source topic
rebuilt: when the sourceTopic or the referenced topic is newer than the topicDependency file

sourceMap2mapDependency

input: 1 sourceMap and every map which it conrefs
action: captures the map dependencies
output: one mapDependency file per map referenced by the source map
rebuilt: when sourceMap or the referenced map is newer than the mapDependency file

sourceTopic2resolvedTopic

input: 1 sourceTopic and zero or many contentDependency files
action: populate conrefs with their contents and non-topic xrefs without inline hotspot text
output: 1 resolvedTopic
rebuilt: when sourceTopic or any contentDependency file is newer than the resolvedTopic
note: The process could verify that the conreffed or xrefed fragment changed

resolvedTopic2titledTopic

input: 1 resolvedTopic and zero or many topicDependency files
action: populate topic xrefs and links without provided hotspot text
output: 1 titledTopic
rebuilt: when the resolvedTopic or any topicDependency file is newer than the titledTopic
note: The process could verify that the topic title changed

sourceMap2resolvedMap

input: 1 sourceMap and zero or many mapDependency files
action: populate conrefs with their contents
output: 1 resolvedMap
rebuilt: when the sourceMap or any mapDependency file is newer than the resolvedMap
note: The process could verify that the conreffed fragment changed

resolvedMap2titledMap

input: 1 resolvedMap and many resolvedTopics
action: populate titles and shortdescs with their contents
output: 1 titledMap
rebuilt: when resolvedMap or a referenced resolvedTopic is newer than the titledMap
note: only the titles of topics that are new to this map or have changed are refreshed

titledMap2mapTopics

input: 1 titledMap
action: split out relationships for each topic
output: one mapTopic file per topic with related-link containing every relationship to another topic with any conditional permutation
rebuilt: when the delta between the previous titledMap and the new titledMap hits a relationship for the topic

titledTopic2titledMapTopic

input: 1 titledTopic and many mapTopics
action: merge the titledTopic with the links from the mapTopics for the topic
output: 1 titledMapTopic
rebuilt: when the titledTopic or any of the mapTopic files for the topic are newer than the titledMapTopic
note: Many different permutations of mapTopic files can be combined with the titledTopic

titledMapTopic2filteredTopic

input: 1 titledMapTopic and a DITA values file
action: filter the topic contents and relationships
output: 1 filteredTopic
rebuilt: when the titledMapTopic or DITA values file is newer than the filteredTopic

titledMap2filteredMap

input: 1 titledMap and a DITA values file
action: filter out the map
output: 1 filteredMap
rebuilt: when the titledMap or DITA values file is newer than the filteredMap

filteredTopic2formattedTopic

input: 1 filteredTopic and a DITA values file
action: generate the formatted HTML or FO topic including flagging
output: 1 formattedTopic
rebuilt: when the filteredTopic or the flagging in the DITA values file are newer than the formattedTopic

filteredMap2formattedMap

input: 1 filteredMap
action: generate the formatted help TOC
output: 1 formatted help TOC
rebuilt: when the filteredMap is newer than the formatted help TOC

filteredMap2formattedBook

input: 1 filteredMap, many formattedTopics, and a DITA values file
action: generate the formatted book including flagging
output: 1 formatted book
rebuilt: when the filteredMap, the formattedTopics, or the flagging in the DITA values file are newer than the formatted help TOC
note: Assumes (naively?) that basic layout of the topic content such as blocks can be created in FO prior to page layout and heading sizing

Note that the resolvedTopic and resolvedMap are deployable for transcoding on request. The actions subsequent to resolution should be possible to perform on request.

The tricky parts are the deltas.

It would be useful to maintain titles and short descriptions in a single registry to minimize file opens.