Skip to main content

Job-Execution Scheduling

The Maintenance window setting of the target host is checked whenever an action launches a job to perform an operation that writes changes to the host. If the setting is set to Never, the job cannot be created and the action fails immediately. If a job is created during a valid execution window, the job is queued for immediate execution (as long as mediated mode is off and host is in managed state). Otherwise, jobs stay in pending status until a valid execution window is reached. During this time, there are no timeouts; the job can wait indefinitely for the window to be reached. Once the execution window of a job is reached, the maintenance window is verified again. If it has not been changed in the meantime, the job is moved to the execution queue. Note that jobs that do not write changes to hosts (such as host scans) are not limited by maintenance windows.

Key Manager polls the queued jobs every 5 seconds, sorting the jobs primarily by priority (lower numerical value means higher priority) and secondarily by creation time. When a job is to be run, Key Manager assigns the job to a worker process for execution. At the same time, any other jobs queued for the same host as this job are also assigned to the worker process. The worker process receiving the jobs sorts and runs the jobs according to the same criteria (priority, creation time).

Whenever jobs reach their execution windows, and whenever the Key Manager system has available system resources, Key Manager starts new worker processes that handle job execution. Each worker process is responsible for running management jobs on a particular host in the managed environment. If the Key Manager system is operating at full capacity, Key Manager finishes the currently-running worker processes before starting new ones. In other words, due to the nature of worker processes, this means that Key Manager runs all the jobs for a certain host before terminating the worker process responsible for that host. Worker processes for other hosts are created and started once system resources become available.

The following host settings affect job priorities:

  • Scan job priority: The priority with which Key Manager runs scan jobs. Smaller value means higher priority. Key-activity-scan jobs are run with a slightly lower (+2) priority.

  • Key action job priority: The priority with which Key Manager runs key-management jobs (all jobs except scan jobs) on hosts. Smaller value means higher priority.

There are two settings that affect job priorities. The priority of scan jobs is set using the Scan job priority setting, and the priority for key-management jobs is set using the Key action job priority setting. In the command-line client, the corresponding settings are scan_priority and action_priority under the host category. The default scan priority is 10 and the default key-action priority 5. These can be changed by modifying the global setting, or overridden by host or host-group-specific settings. This allows specific important hosts or host groups to execute their jobs before others.

The job priorities may also be modified by prioritizing them after the job has already been created. To do this, navigate to the Logs→Jobs page, and perform a Prioritize action on the target job(s). Jobs with smaller priority numbers are run before jobs with bigger priority numbers.

The priority of a job may also be specified with the command-line client command set-job-priority (described in set-job-priority). For example:

$ ssh-mgr-client set-job-priority \
-F 'hostname=server.example.com&&job_type=scan*' -p 1

If a job cannot be executed for some reasons, such as due to the target host being offline at the time, Key Manager schedules the job to be run at a later time. Jobs are retried at an increasing interval based on the number of retries, with the first retry scheduled after 60 seconds, the next after 120 seconds, and so on (60, 120, 240, 480... 30720). 10th and later retries are performed at 30720-second intervals. The maximum number of retries performed for a job is set using the host setting Maximum amount of retries for a job.

The maximum number of retries can also be set using command-line client commands that allow you to modify host settings (set-global-setting, set-host-group-setting, and set-host-setting). For example, setting the maximum number of retries for a certain host using set-host-setting (described in set-host-setting):

$ ssh-mgr-client set-host-setting -I "hostname=server.example.com" \
-d "category=host,name=max_job_restarts,value=20"

Jobs for adding and removing temporary authorizations are created before such authorizations are to be added or removed:

  • Jobs for adding temporary authorizations are created immediately after the associated access request is approved for execution. The created jobs are scheduled to run at the beginning of the validity period.

  • Key Manager periodically polls for authorizations that are about to expire (default polling interval is 10 minutes). If any temporary authorization is found to expire within 1 hour or less, Key Manager creates jobs for removing such authorizations. The created jobs are scheduled to run at the end of the validity period.

Note that jobs for adding and removing authorizations are subject to standard Key Manager job-scheduling behavior. The actual time when temporary authorizations are added/removed may be delayed (for example, if the Key Manager system is busy running higher-priority jobs, or if the target host is unavailable). In all cases, the start of a validity period specifies the earliest moment when the authorization may be added. Similarly, the end of a validity period specifies the earliest moment when the authorization may be removed.

note

Maintenance windows are validated against the clock of the Key Manager Database.

For agent-based hosts, all jobs are run only during agent connections. Jobs for agent-based hosts set with long agent-connection intervals may be significantly delayed.