Checklist: Data Missing in Service Engine

Checklist: Data Missing in Service Engine

Time estimate: 60 minutes

Please go through all the points in the following checklist. If necessary, confirm with IT Department any information you cannot verify before contacting your local COPA-DATA Representative.
Checklist usage: #. [Quick hints] (point refers specific module/product) Point description.
Terminology:
Service Engine folder = RT folder + Data folder (*).
      RT folder = Service Engine files created by Engineering Studio.
      Data folder (computer name) = Files created by Service Engine during operation (alarms, events, archives, etc.).

(*) Data folder is not necessarily a subfolder of Service Engine folder. Its location can be amended in the project properties.
  1. Screenshot and followed instructions of error/warning messages shown.
  2. Data folder is not in a removable or a network device.
    3. It is recommended that the data is recorded locally and backed up externally (if necessary).
  3. License includes the modules used to write/access data in the format desired.
    4. You can find all license details of zenon Software Platform products in License Manager (in zenon Startup Tool > Tools).
  4. Service Engine ran, could write/access data in data folder and was closed correctly before.
    1. In Service Engine Data folder, ARX, ARS, ARI, CEL/CEL.BIN, AML/ALARM.BIN files are not configured as "Read-only". Check the files attributes in properties.
    2. [Task Manager -> Performance] Operating System has free memory.
    3. There is available disk space or folder quota to write data in Service Engine folders.
      4. You can use the system driver '[System information] Service Engine folder not available' variable to audit the data folder availability.
      Other system driver variables in '[Hardware resources]' can also show the available memory and the free disk space of the partition where the Service Engine and Export folders of a project are located.
    4. Service Engine has enough reserved memory to read and process data.
      5. The amount of memory reserved for archive data processing in Service Engine can be set in project configuration file (project.ini).
      To make changes permanently, you can modify the project.ini of the project-editor files:
      1. In Engineering Studio, click in the project name (root) and press ALT+CTRL+E keys. Navigate to the subfolder\zenon\system and open the project.ini file.
      2. Add the section below (if not present yet), compile new Service Engine files and restart Service Engine. 
      [ARCHIV]
      SPEICHER=X    ;(X kilobytes, e.g. 2000000 = 2 GB)

      Hint: System driver variable '[Historian] Memory for Historian' shows the current memory reserved for archive data processing in Service Engine.
  5. Operator inserted a valid filter (e.g. timeframe, variables name, etc.). Data is also not read/shown when not using any filter.
  6. Searched and followed error instructions in LOGs related with the components/processes and modules involved in data-acquisition (e.g. zenon Drivers), data-storage and data-reading (e.g. CEL/AML/Archive).
    7. All zenon components write in the background messages to joint log file(s) saved in '%programdata%\COPA-DATA\LOG' folder.
    These files can be read and configured with Diagnosis Viewer.

    You can modify the logging level of a component by clicking 'Client Configuration' in 'Settings' menu and after select the respective process name.
    Attention: Set the logging parameters wisely as the time-frame of the LOGs decrease when increasing the detail of the logs written.

    You can also find LOG files in SIC in 'Platform > Versions > Version XXXX > Directories > LogDirectory' folder.
    Service Engine writes log messages in regard zenrt32.exe stability in case of errors or warnings.
    You can find these entries in Diagnosis Viewer by filtering the column Function name = 'CZenonrtApp::MonitoringThread'.

    Some of the most relevant performance entries are:
    - Memory: “Total available memory fell below warning/error level to [xxx]%”
    - CPU: Total CPU-Load exceeded warning/error level to [xxx]%”
    - Service Engine not responding for XXX ms: "100ms Timer has not cycled for [xxx]ms”
    - Handles: "(%) “Handles exceeded error level to [xxx]”.

    In case Service Engine closes unexpectedly, zenon Drivers write in the logs "RPC error!" as they can no longer communicate with zenrt32.exe.
    1. There are no driver communication errors logged and process driver variables status/timestamps are legit.
    2. Time synchronization is configured and working. All devices in the system use the same time source (external or zenon).
    By default, zenon Time Synchronization is enabled in all computers in zenon network.
    zenon Time Synchronization can be disabled when using an external time source.
    For this, in [NETZ] section in zenon configuration file (zenon6.ini) (path: %CD_SYSTEM) add the line 'TIMESYNCH=0' .
    The same configuration must be applied to all zenon computers in the network.
  7. Other Service Engine modules also cannot access the data.
    8. Recall, you can use different screens/modules to access data in Service Engine. For example,
    Archive data: Archive Revision + Extended Trend + Report Viewer/Generator screens.
    Alarms and Events: Alarm Message List + Chronologic Event List + Report Viewer/Generator screens.
  8. Searched in Knowledge Base if a previous issue has been addressed.
  9. In File Explorer, the Data folder contains files created/modified at the time of the missing data.
  10. [Event Viewer] Operating System Event-logs do not show critical/errors/warning events in terms of network, time, hardware or software and do not indicate that the system revert to a restore point.
  11. Local/Group Policies or user rights do not restrict access (write - writing data | read - reading data) to the data folder. Confirm with IT department if there were no changes implemented recently.
  12. Antivirus software logs do not report actions to prevent writing/accessing data. Antivirus software is disabled.
  13. Data is missing following a network problem where both servers ran in parallel as process leading server.
    14. In redundant server projects constellation, following a network problem it may happen that both Service Engine servers run as leading servers independently. This means that some data can be overwritten when the server with data sovereignty takes the leading role. To prevent such scenario, you can use  Redundancy Management Tool (in zenon Startup Tools > Tools) to monitor the network connection in each server and close/start automatically the local Service Engine in case a network problem is detected/resolved.
  14. [zenon Startup Tool > Applications > Network configuration] Try to increase the ‘Network module timeout’ when trying to access data from a Service Engine client (the configuration should be identical in all zenon network computers).

If the problem persists after completing this checklist and followed corrective actions, please contact your local COPA-DATA Representative providing a SIC-Reduced Report from the target computer(s) and the result of this checklist – please include any additional information or comments related with the points addressed you find relevant.

System Information Collector is a standalone COPA-DATA application that collects relevant data about the Operating System and zenon Software Platform for troubleshooting purposes. SIC is installed with zenon and can be started from zenon Startup Tool (Tools).