Symptoms

A program run by adTempus fails immediately with an exit code of 128 or -1073741502. If the program normally produces logs or other output, no output is created and the program appears to not have actually run.

On Windows Server 2003 and later, the following message may be logged in the System Event Log at the same time that this problem occurs:

Event Type: Warning
Event Source: Win32k
Event Category: None
Event ID: 243
Date: Date
Time: Time
User: N/A
Computer: ServerName
Description: A desktop heap allocation failed.

The problem generally occurs with jobs using a "Hidden" User Interaction setting or steps using a "Hidden" or "Capture Console" Window Mode.

Scheduled programs may continue to fail with exit code 128 or -1073741502 until some programs currently being run by adTempus finish or the adTempus service is restarted.

Cause

The exit codes 128 or -1073741502 are returned to adTempus by Windows and indicates Windows was not able to initialize or execute the program because there is not enough desktop heap memory available for the program to run.

This problem may occur if you have a large number of jobs running simultaneously in "Hidden" interaction mode under the same user account, and/or one or more of the programs being run uses a large number of user interface or GDI resources.

More Information

All Windows programs execute on a virtual windowstation and desktop. If a program runs on the "interactive" windowstation and desktop, it is visible to the user and can accept user input. In addition to the interactive windowstation, there may be any number of non-interactive windowstations and desktops. Programs that are associated with these non-interactive desktops are not visible to the user.

When adTempus runs a program, the program may be associated with an interactive or non-interactive desktop:

  • If the Window Mode setting for the step is "Hidden" or "Capture Console," the program runs on a non-interactive desktop. Otherwise,
  • If the User Interaction setting for the job is "Console" or "User's Session", the program runs on the interactive desktop.
  • If the User Interaction setting for the job is "Hidden," the program runs on a non-interactive desktop.

All non-interactive programs run by adTempus under a single user ID run on the same non-interactive desktop. For example, if there are 20 jobs running at the same time under the same user ID, they will all run on the same non-interactive desktop. If there are other jobs running under different user IDs, they will run on different desktops.

Each windowstation/desktop has a fixed amount of desktop heap memory (which is unrelated to the amount of system memory) that is shared by all programs running on that desktop. By default, the interactive windowstation/desktop has 3MB of desktop heap memory, and each non-interactive windowstation/desktop has 512KB of memory. The desktop heap memory is used for user interface objects such as hooks, menus, strings, and windows.

The amount of desktop heap allocated to non-interactive desktops is relatively small, because most programs that run on non-interactive desktops do not display user interfaces, and therefore consume only a very small amount of desktop heap. However, adTempus allows you to run programs that might not have been designed with non-interactive execution in mind, and such programs may use larger amounts of desktop heap.

For example, you might be calling Excel to extract and format data from a data source. Even though you can use scripts and macros to automate Excel so it can work without user interaction, Excel was designed as an interactive application, and still displays its full user interface when you do this. Displaying the user interface consumes memory from the desktop heap.

If you run several such programs at the same time, or a single program that uses a large amount of desktop heap memory, it is possible to use all of the desktop heap memory available for the windowstation/desktop. Once this heap memory is exhausted, the executing programs may not function properly, because they will be unable to create new windows and other user interface objects. Additionally, adTempus will not be able to start any more programs on that desktop, because a small amount of desktop heap memory is required for each program.

When this point is reached, the program fails to initialize when adTempus tries to start it, and the exit code is reported.

Resolution

There are three possible resolutions for this problem:

Restart the adTempus Service

To allow adTempus to begin running programs again quickly, you can restart the adTempus service. After adTempus is restarted, it will not continue to execute programs on the non-interactive windowstation and desktop it was previously using (and for which the desktop heap memory has been exhausted). It will create a new non-interactive windowstation and desktop, and will be able to continue executing programs.

In some cases the initial heap exhaustion may have been the result of a rare combination of circumstances, or of a program using an unusually large amount of heap memory. If this is the case, you may not need to take any action beyond restarting the service.

However, if the problem occurs again, you will need to proceed to one of the following resolutions.

Identify and Reconfigure the Applications Causing the Problem

If you can identify which applications are causing the problem, you may be able to change their scheduling so they do not run at the same time. For example, you may notice that you encounter this problem when you run two reporting applications at the same time, but if you change their scheduling so they don't run at the same time, the problem does not occur.

If you cannot identify the applications causing the problem, or it is not feasible to reschedule them, you will need to increase the amount of desktop heap that is available, as described below.

If you are not able to increase the non-interactive heap size sufficiently to allow your job to run, you will need to consider again whether jobs can be rescheduled in such a way that the problem does not occur. Alternatively, if the problem is with a single job, you can change the job and step to run interactively (set the User Interaction mode for the job to "Console," and the Window Mode for the step(s) to "Normal"). This will cause adTempus to execute the program on the interactive windowstation/desktop, which has more heap memory available to it. However, this approach has its own drawbacks, any user connected to the computer's console session will be able to see and interact with the program.

Increase the Desktop Heap Memory

Warning: Making the following changes could have serious adverse side effects. If you do not understand what needs to be done and the potential consequences, please contact us for assistance before making these changes. See Microsoft Knowledge Base Article 184802 for additional information about the settings discussed below.

As discussed above, by default a non-interactive windowstation/desktop is allocated 512KB of memory by Windows. It is possible to increase this memory by changing a setting in the Registry.

However, increasing this memory allocation involves a tradeoff. Windows has a fixed buffer (either 20MB or 48MB, depending on operating system and configuration) from which all windowstations and desktops are allocated. Therefore if you increase the amount of memory allocated to non-interactive windowstations/desktops, you decrease the number of windowstations and desktops that can be created on the system. Because other applications and services may be using non-interactive windowstations and desktops, if you increase the heap size too much, it is possible that you will run out of space to allocate additional windowstations, which may cause adTempus or other applications or services to fail.

Therefore you need to increase the desktop heap memory allocation by the smallest amount possible.

To change the heap size, run the Registry Editor and go to key

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems

The "Windows" value under this key will appear similar to the following:

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows  SharedSection=1024,3072,512 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3    ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off  MaxRequestThreads=16

The values following "SharedSection=" control desktop heap allocation, and the third value ("512" in the example above) controls the non-interactive desktop heap, which is what needs to be increased.

Begin by increasing this value by 512, which adds an additional 512KB of heap memory. In the example above, you would change the value to read:

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072,1024 Windows=On SubSystemType=Windows  ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16

Note: If the third SharedSection value is missing from your current settings, simply add ",1024" after the existing values.

After making this change, you must restart the computer for it to take effect.

If you continue to encounter the problem after making the change, you can try increasing the heap value further. If you increase it too far, you will hit the systemwide buffer limit, causing other problems. On Windows Server 2003 and later, the following warning will be reported in the System Event Log if you reach this limit:

Event Type: Warning
Event Source: Win32k 
Event Category: None 
Event ID: 244
Date: Date
Time: Time
User: N/A 
Computer: ServerName
Description: Failed to create a desktop due to desktop heap exhaustion.

If this occurs, you need to decrease the heap memory size.

References