Checklist: Client/Standby cannot establish a reliable connection with Service Engine Server

Checklist: Client/Standby cannot establish a reliable connection with Service Engine Server

Time estimate: 45 minutes

Please go through all the points in the following checklist. If necessary, confirm with IT Department any information you cannot verify before contacting your local COPA-DATA Representative.
Notes
Checklist usage: #. [Quick hints] (point refers specific module/product) Point description.
Info
Terminology:
zenon network = All Service Engine computers in the network
Primary Server = Process Leading Service Engine computer
Client(s) = Standby + Client Service Engine computer(s)
  1. Before contact the support, document screenshot of error/warning messages shown and followed instructions.
  2. Check that only zenon network is affected so that there is no reliable communication between Primary Server and all/some clients. If not only zenon is affected then it is for sure an IT issue.
  3. Possible symptoms in zenon: Service Engine on clients show blue-dots in the top right corner of screen elements (status N_UPDATE=1). 
    Or as well zenon 'Server 1' and 'Server 2' are both working as Primary Server, and not as Primary and Standby. Or the redundancy switch function has no effect. 
  4. zenon Server(s) started the correct project. Confirm the project name and project GUID, when starting the project from Engineering Studio.
  5. In Engineering Studio there were no changes in project settings for Network. Or if edited and (even if not changed) left not by Esc key, then Servers' Service Engines project were updated by restart, not by reload.
  6. zenon Drivers still communicate with field devices so that there are no red-dots shown in the top right corner of screen elements in the Primary Server (status INVALID=1).
  7. Network Service process (zenNetSrv.exe) is running with identical configuration (default: TCP Port = 1100 and IPv4) in all computers in zenon network.
    8. Notes
    Network Service settings can be consulted or amended in  'zenon Startup Tool > Application > Listening ports > Services: Network Service'.
    Warning
    All computers must have the same password when using encrypted network communication.
    You can insert the password in zenon Startup Tool > Application > Network configuration.
  8. Server names in Project > Network settings use computers hostname or FQDN (Fully Qualified Domain Name).
    9. Alert
    Do not insert a computer IP address, localhost or 127.0.0.1 as server name.
  9. On all computers, in Service Engine, the SYSDRV variables with current Process Server and Standby names show FQDN, not just hostname (even if project settings are hostnames, not FQDN). If not, check that operating system knows correctly the 'Primary DNS Suffix', and not only the Ethernet adapter specific DNS suffix; even on PCs with only one Ethernet adapter.
  10. Network name resolution works between all computers in zenon network.
    11. Notes
    In Windows, you can use cmd ipconfig /all to check that 'Primary DNS suffix' is not empty.

    Ping commands between Server and Clients are successful in both directions using computers hostname/FQDN.
    Idea
    You can use nslookup (stands for "Name Server Lookup") to query the local DNS Server to obtain the name or IP address mapping of the Domain Name System (DNS).
    For example, the Command Prompt command 'nslookup IP' or 'nslookup hostname', returns the DNS record of a specified IP/hostname (local or other machine) - the result should be identical in every computer in the network.
    1. In domain networks, when executing the command 'ping -a [hostname]', the resolved computer name always includes the DNS suffix (FQDN).
      2. Alert
      For example,
      In Server1 executing the command: ping -a Server1
      In Server2 executing the command: ping -a Server1

      In Server1 executing the command: ping -a Server2
      In Server2 executing the command: ping -a Server2

      The resolved name of the target computer of the ping command always includes the hostname and DNS suffix (FQDN) of the computer. If not, and to flush the DNS does not solve, as workaround an admin can enter the IP address and the FQDN of the computers in the Hosts files (C:\Windows\System32\drivers\etc\hosts) of the servers and all clients.
  11. Primary Server and Client(s) network connection is established.
    12. Notes
    You can use Windows Resource Monitor to know more details about running processes.
    In the Primary Server (and clients), open Windows Resource Monitor >  Network tab and select zenNetSrv.exe process.
    The TCP Connections window should give you a clear picture about the connections established between Primary Server and Client computers.
    The Listening Ports output should match the configuration of the Network Service in zenon Startup Tool and be identical in all computers in zenon network.
    Idea
    Alternatively, you can execute the command in Command Prompt (run as administrator) ‘netstat -a -b | findstr :1100’.
  12. Searched and followed error instructions in LOGs related with Network Service (zenNetSrv.exe) in Primary Server and in Clients with communication problems.
    13. Idea
    All zenon components write in the background messages to joint log file(s) saved in '%programdata%\COPA-DATA\LOG' folder.
    These files can be read and configured with Diagnosis Viewer.

    You can modify the logging level of a component by clicking 'Client Configuration' in 'Settings' menu and after select the respective process name.
    Attention:  Set the logging parameters wisely as the time-frame of the LOGs decrease when increasing the detail of the logs written.

    You can also find LOG files in SIC in 'Platform > Versions > Version XXXX > Directories > LogDirectory' folder.
  13. Time synchronization of all computer clocks is configured and working.
    1. All devices in the system use the same time source (external or zenon).
    2. Time difference between computers is shorter than five (5) seconds.
    Info
    By default, zenon Time Synchronization is enabled in all computers in zenon network. This must be disabled in redundant Network with a non-dominant server, use external time source instead.
    Generally, zenon Time Synchronization shall be disabled when using an external time source (note: Windows, by default, runs a NTP-client synchronizing the clock).
    For this, in [NETZ] section in zenon configuration file (zenon6.ini) (path: %CD_SYSTEM) add the line 'TIMESYNCH=0' .
    The same configuration must be applied to all zenon computers in the network.
  14. [Event Viewer] Operating System event-logs do not report any problem related with hardware, network or time synchronization at the time of the problem.
  15. [zenon Startup Tool -> Applications -> Network configuration] Increased zenon ‘Network Timeout’ if the number of files to synchronize between Primary and Standby servers is large.
    16. Alert
    It will take longer the standby to recognize a server failure and upgrade himself to primary. It is also required more memory in the standby to buffer data.
  16. Searched in Knowledge Base if a previous issue has been addressed.

If the problem persists after completing this checklist and followed corrective actions, please contact your local COPA-DATA Representative providing a SIC-Reduced Report from the target computer(s) and the result of this checklist – please include any additional information or comments related with the points addressed you find relevant.

Info
System Information Collector is a standalone COPA-DATA application that collects relevant data about the Operating System and zenon Software Platform for troubleshooting purposes. SIC is installed with zenon and can be started from zenon Startup Tool (Tools).