Checklist: Process is not responding

Checklist: Process is not responding

Time estimate: 60 minutes

Please go through all the points in the following checklist. If necessary, confirm with IT Department any information you cannot verify before contacting your local COPA-DATA Representative.
Checklist usage: #. [Quick hints] (point refers specific module/product) Point description.
Terminology:
Service Engine folder = RT folder + Data folder (*).
      RT folder = Service Engine files created by Engineering Studio.
      Data folder (computer name) = Files created by Service Engine during operation (alarms, events, archives, etc.).

(*) Data folder is not necessarily a subfolder of Service Engine folder. Its location can be amended in the project properties.
  1. Screenshot and followed instructions of error/warning messages shown.
  2. Process ran and was closed correctly before.
  3. Searched and followed error instructions in LOGs related with the process-executable not responding.
    4. All zenon components write in the background messages to joint log file(s) saved in '%programdata%\COPA-DATA\LOG' folder.
    These files can be read and configured with Diagnosis Viewer.

    You can modify the logging level of a component by clicking 'Client Configuration' in 'Settings' menu and after select the respective process name.
    Attention: Set the logging parameters wisely as the time-frame of the LOGs decrease when increasing the detail of the logs written.

    You can also find LOG files in SIC in 'Platform > Versions > Version XXXX > Directories > LogDirectory' folder.
    Service Engine writes log messages in regard zenrt32.exe stability in case of errors or warnings.
    You can find these entries in Diagnosis Viewer by filtering the column Function name = 'CZenonrtApp::MonitoringThread'.

    Some of the most relevant performance entries are:
    - Memory: “Total available memory fell below warning/error level to [xxx]%”
    - CPU: Total CPU-Load exceeded warning/error level to [xxx]%”
    - Service Engine not responding for XXX ms: "100ms Timer has not cycled for [xxx]ms”
    - Handles: "(%) “Handles exceeded error level to [xxx]”.

    In case Service Engine closes unexpectedly, zenon Drivers write in the logs "RPC error!" as they can no longer communicate with zenrt32.exe.
  4. [Task Manager > Details] Operating System and other applications are responsive.
  5. It is (not) possible to establish a:
    1. Periodicity or the exact time when the process freeze occurs.
    2. Sequence of actions and/or the functions executed prior to the problem.
  6. Changing process/component configuration can avoid the freeze (e.g. narrower time/data filter, avoid execution of certain function, etc.).
  7. Process cannot recover from the freeze so that it must be forcibly closed.
    8. Some remarks:
    - It is important to distinguish short continuous freezes from one single continuous freeze.
    - A process can be temporarily unresponsive when performing heavy operations.
    -  Consider different strategies to balance and flatten the overall load of the system by splitting functionality through different computers (e.g. Process Gateway runs on a specific Service Engine Client) and different moments (e.g. heavy duty tasks such as the ones requiring data processing are executed a long a certain period and not at the same time).
    - Consider to acquire better hardware.
  8. Cyber security software logs do not report actions to prevent the execution of the process. Cyber-security software is disabled.
  9. Process also freezes when zenon API is disabled.
    10. zenon API can be switched off by editing the respective sections in zenon configuration file (zenon6.ini) located in %CD_SYSTEM% folder:
    [VBA]
    [VSTA]
    [ADDINS]
    EIN=0
    ON=0
    ON=0
  10. There are not network issues that can affect process stability.
    11. High network latency, malformed packages and other unwanted conditions may cause unexpected behaviours on processes relying on communications.
  11. [Task Manager > Performance] Determined Operating System overall load when idle (process not running) and during process execution.
    1. CPU utilisation level per logical processor (core).
    2. Threads and Handles count.
    3. Memory load.
  12. [Task Manager > Performance] When the process is unresponsive:
    1. A single CPU core shows a 100% load (or close).
    2. There is no free memory available (or very small percentage).
    3. Handles and Threads order of magnitude is higher than when the process was fresh started (also for the total Operating System count).
  13. (Service Engine) Process also becomes unresponsive after the project RT folder has been recreated.
    1. Closed Service Engine.
    2. Renamed (backup) the original RT folder.
    3. Re-created (and remote transported) all Service Engine files.
    4. Restarted Service Engine.
  14. (Service Engine) Process also becomes unresponsive after the project Data folder has been recreated.
    1. Closed Service Engine.
    2. Renamed (backup) the original Data folder.
    3. Restarted Service Engine.
  15. Searched in Knowledge Base if a previous issue has been addressed.
  16. [SIC-Full] Collected process crash full dump file in case the component closes unexpectedly and produces an unhandled exception.
    17. Crash full dumps can be generated by adding a [DEBUG] section in zenon6.ini:
    [DEBUG]
    MINIDUMPTYPE=10

    Future full dumps will be saved in the ‘%programdata%\COPA-DATA\zenonXXX’ folder.

If the problem persists after completing this checklist and followed corrective actions, please contact your local COPA-DATA Representative providing a SIC-Full Report from the target computer(s) and the result of this checklist – please include any additional information or comments related with the points addressed you find relevant.

System Information Collector is a standalone COPA-DATA application that collects relevant data about the Operating System and zenon Software Platform for troubleshooting purposes. SIC is installed with zenon and can be started from zenon Startup Tool (Tools).