Speeding up the Export Process

What Slows down the Export

Especially on large websites, changing a file sometimes has the effect that the Template Engine considers many other files obsolete because they depend on the file that was modified. As a consequence, the export process takes more time because many files have to be exported again.

The following actions considerably slow down the export:

  • Changes to layout files. They cause the partial hierarchy, in which the layout is effective, to become obsolete and to be exported again. This is the case even if the layout files are not used during the export.

  • Renaming folders. This also has the effect that all files in this partial hierarchy and additionally all files containing links pointing to a file in this hierarchy need to be exported again since the path of the files contained in the folder has changed.

The impact of the following actions is weaker:

  • Changes to the file name extension. This causes all files containing links to the file in question to be exported again.

  • Changing the size of images. This also causes all files containing links to the file concerned to be exported again because the links potentially contain image dimensions.

  • Changes to the suppressExport field. References (such as links) to the file concerned are made visible or not by the export process, depending on the value of this field. Changing its value therefore causes the files containing the references to be invalidated.

  • If files are created or deleted, all files containing tables of contents of the folders concerned become obsolete.

  • Changes to most of the version fields invalidate the file itself as well as all other files from which field values are read while they are exported.

Approaches for Reducing the Export Time

If you include content statically using includetext, it is recommended to include the content dynamically instead using insertvalue = "dynamiclink". Furthermore, the keys hasSuperLinks, superLinks, and superObjects should be used with caution in layouts because they have the effect that all other files become dependent on the queried file.

Thus, the time needed for an export does not result from the number of files alone but also from the complexity of the layout structure and the calculation expense produced for each exported file. This calculation expense mainly results from the frequency and the type of the queries contained in the layouts. For example, it is much more time-consuming to generate file or link lists than to query individual field values such as title.

Analyzing Dependencies

Whether dependencies are the cause of exports running slowly can be determined in the following way.

  • In the Content Manager, execute the command incrExport reset and transfer your content to the Template Engine using job withName systemTransferUpdates exec.

    Attention: While this command is executed, update records must not be generated. The incrExport reset command must only be executed in single mode and only if it is guaranteed that no write access to the data takes place. Otherwise the exported data will be incomplete or destroyed. Therefore, please make sure that the CM is not running as a server and no other CMs are executed in single mode while this command is being executed.

  • After all update records have been transferred, execute the command app export in the Template Engine.

  • Immediately after the export has completed, check the usesAll dependencies that were generated. You can use the command statistics compute name all for this and analyze the Template Engine’s log file (each time a dependency of this type is generated, a warning is written to the log).

  • Try to determine why the files concerned depend on many other files. For this purpose, the value of the system configuration entry tuning.export.usesAllThreshold can be increased (the Template Engine must be restarted after that) and afterwards some of the files concerned can be updated (obj touch and obj updateCache). If, as a result, no usesAll dependencies are generated, obj withId id get dependencies will show the IDs of the files on which the files concerned depend.

    If you would like to determine the layout file and the NPSOBJ instruction that caused the dependency, first increase in the server.log.logger system configuration entry named info the value for the level attribute to 3. Then restart the Template Engine. Update one of the files concerned using obj withPath path updateCache. In the info.log you can now find the export details mentioned above for this file.

To reduce the number of dependencies, please proceed as described above, i. e. reduce the complexity of your layout structure and the calculation expense in the layout files.