adTempus.jobhost.exe / kernelbase.dll error message - Community

Ken T (33 posts)

January 30, 2023 06:56 PM

Accepted Answer

Hi Bill,

Would you have any idea why we would be getting this message?

I think the jobs are most if not all are Powershell script jobs.

Faulting application name: adtempus.jobhost.exe, version: 4.6.0.0, time stamp: 0x5c1ba5ab
Faulting module name: KERNELBASE.dll, version: 6.3.9600.20512, time stamp: 0x62cdfc6e
Exception code: 0xc0000142
Fault offset: 0x00000000000ed1b0
Faulting process id: 0x376c
Faulting application start time: 0x01d9350f5b3a7691
Faulting application path: D:\Program Files\Arcana Development\adTempus\4.0\adtempus.jobhost.exe
Faulting module path: KERNELBASE.dll
Report Id: 9948c544-a102-11ed-8104-005056ac32d8
Faulting package full name:
Faulting package-relative application ID:

Bill Staff (604 posts)

January 31, 2023 12:52 PM

Accepted Answer

The jobhost process is the process that adTempus launches to run scripts within a job. It looks like it is failing at initialization, which could mean a permission problem or it could mean that there is a shortage of desktop heap memory (could happen if there are a lot of jobs running, or if there are programs using a lot of GUI resources).

The easiest first thing to try is to restart the computer and see if the problem goes away.

Otherwise you will need to open a support case so we can look into this further.

Ken T (33 posts)

January 31, 2023 04:56 PM

Accepted Answer

Thanks Bill.

I have a few questions.

Would checking the C:\Users\*\AppData\Local\Arcana Development\adTempus\ScriptEngine\Cache folder be related at all this error message?

This is an intermittent problem. The same job will run later. Some jobs are scheduled to run 10 or 15 minutes apart.

Will a recycle of the adTempus service be possible instead of a server recycle?

Will upgrading to adTempus 4.7 or later make any difference?

When you say heap memory where would I check on the adTempus server? I see the adtempus.jobhost.exe entries listed in Resource Monitor. Though I see several memory related columns,Commit (KB), Working Set (KB), Shareable (KB), Private (KB). Is there any one column that I should focus on? I also the corresponding processes under the Processes tab, showing adTempus script host. The PID values correspond to entries in the Resource Monitor.

Those entries should appear and disappear when the jobs complete, right?

Could you explain a little more about the file entries in the cache folder? Are they safe to delete if no jobs are running?

Also, it's our adTempus job server, so I don't think there are much GUI resource requirements.

Is there a certain way to write Powershell scripts that are friendly with memory when exceptions occur?

Thanks for all your help. I want you to know over the years, I've been a strong advocate of adTempus and it's capabilities! I think it's a great product!

Ken

Ken T (33 posts)

January 31, 2023 05:17 PM

Accepted Answer

Also, I just noticed there are 11 adTempus script host entries in Task Manager, but don't see any active jobs running in the adTempus Console via the Job Monitor > Show Active.

What should I make of that?

Bill Staff (604 posts)

February 1, 2023 02:50 PM

Accepted Answer

See this article for a discussion of desktop heap memory. The initialization failure discussed there is what is happening with the script host.

The most likely cause is that you have a lot of scripts running in the same login session and they are exhausting the heap. We would have to dig more into what the scripts are doing to figure out why this would be. See if you spot any pattern to what time of day it's happening, what scripts are running, etc.

Restarting the adTempus service will probably resolve this temporarily as it will end the existing logon session(s) and force adTempus to start using a new one.

For a long-term fix you may need to increase the non-interactive desktop heap size as discussed in that article.

The cache folder is unrelated to the problem. When you run a script written in C# or VB.NET, adTempus compiles the code into a .NET assembly, which is cached here to be used next time the script is run so that it doesn't have to compile the code every time.

Each instance of the jobhost process runs 1 script at a time. After it finishes running the script, it stays running for about 5 minutes so it can be reused if there is another script to be run. Each host only runs scripts for a single user, so you will see several instances if there were several scripts running at the same time, or successive scripts running under different users. This is not likely to be causing a problem but you can reconfigure adTempus to not reuse the processes and see if that helps: go to the Advanced Server Options window and locate the "JobHost:IdleShutdownDelay" option. Set its Current Value to "0". After you do this, the host process should exit more or less immediately after it finishes running a script.

I don't see that we have changed anything in 4.7 or later that should affect this.

Ken T (33 posts)

February 1, 2023 04:28 PM

Accepted Answer

Bill,

Thanks for all your incredible insight!

In terms of memory, the server has about 3GBs in use of 8GB. We'll keep an eye on it.

I have just a frew of more questions with regards to script and jobhost instances.

Does the notion of the script stored in adTempus or stored in external file make a difference?

What about consolidating scripts from a hypothetical twenty jobs into one script and using parameters to call them?

Is there any conflict when running two different scripts under the same user at the same time in two different jobhost instances?

Ken

Bill Staff (604 posts)

February 3, 2023 02:40 PM

Accepted Answer

The amount of total memory on the server is irrelevant. The issue is that Windows only allocates a small amount of memory to the desktop heap of the non-interactive windowstations, and something is causing you to exceed that. The solution is 1) Increase the amount allocated (as discussed in the article I referred you to); 2) Limit the number of jobs per session (see below); and/or 3) Figure out what the script(s) are doing to use desktop heap memory and solve that.

If a script is being run by a Script Execution Task it doesn't matter whether the script is stored internally or externally. It is run the same way. Note that if you were running these scripts using a Program Execution Task instead (calling powershell.exe to run each script rather than using the Script Task, which runs them in the adTempus script host) you would probably eventually get a similar failure when powershell tries to start, and if you have non-script tasks running under the same user account you may see failures with them as well, because the scripts (apparently) are using up the desktop heap memory and this will cause everything else adTempus tries to run in that logon session to fail.

I have no idea if consolidating scripts would make a difference but it's the wrong solution.

You can run as many scripts at the same time as you want, under any number of user accounts. adTempus will start a separate jobhost for each script. If there are several scripts running under the same account they will be run in the same logon session but different host processes.

Possible solution 2: Generally adTempus is going to put all jobs for a user in the same logon session: each time a new job starts, if there is an existing logon session for the user, adTempus will use it. Once there are no jobs running for the session, it gets logged off, and a new session is created for the next job. If you are seeing this problem occur when there are several scripts running at the same time under the user account (but not when there are a smaller number), then the problem is that the jobs are cumulatively trying to use more desktop heap memory than has been allocated for the session. Increasing the size of the allocation (option 1 above) should solve the problem but getting the balance right can be tricky, depending on operating system version and on what else is going on on the server. Also you have to restart the server each time you change the setting.

A possible alternative is to reduce the number of jobs packed into each logon session. For example, if you are seeing the problem when there are 4 jobs running under an account and adTempus tries to start the 5th, you can try limiting adTempus to only 4 jobs per logon session. It would then create a new session for the 5th job. To do this, go to Advanced Server Options and find the "UserManager:MaxJobsPerSession" option. Set the Current Value to the limit you want to use and click OK to save the changes. This change will be applied the next time a job starts (you don't need to restart adTempus).