Zenon 8.20 Redundancy — Standby Server IEC 61850 Driver TCP Connect Errors Causing Variable Freeze

Zenon 8.20 Redundancy — Standby Server IEC 61850 Driver TCP Connect Errors Causing Variable Freeze

  • Zenon Energy Edition 8.20
  • Software Redundancy (Master / Standby)
  • IEC 61850 driver, 50+ IEDs
  • Windows Server 2019, 32 GB RAM, Xeon Silver 4208

    We are running a dual-server redundancy setup. Afte the Standby server started freezing all variable values shortly after startup.

    Symptoms observed:

    • On first startup, Standby server receives values correctly
    • After historian sync completes, all variable values disappear or freeze
    • Some variables partially visible (e.g. Va visible, Vb/Vc missing)
    • Opening trend or alarm pages on Standby triggers another historian sync and causes values to freeze again
    • Both servers show Master in system variables, Standby field always empty — classic split-brain behavior
    • Switching active server restores values on the previously frozen server
    • System restart takes approximately 15 minutes due to large archive size (~40 GB .arx folder, ~1 million files)

    Diagnosis Viewer findings:

    • Hundreds of OSI4 / T5BusExlec850m errors
    • TCP connect errors to all IED IP addresses simultaneously from Standby server
    • IEC 61850 R2 connection errors continuously repeating
    • 529 error lines, queue nearly overflowing

    • Initial assumption was resource pressure due to large historian archive. We renamed the 40 GB .arx folder, recompiled and redeployed the project — system worked correctly on both servers.

    We then attempted to migrate the last 3 months of historian data (customer requirement) in 3 batches. After the 3rd batch was copied, Standby server froze again.

    RAM usage was checked during freeze — only 44% utilized (approximately 14 GB), so memory pressure was ruled out.

    Diagnosis Viewer clearly showed the Standby server IEC 61850 driver attempting TCP connections to all IEDs simultaneously, even while in Standby mode. Since all IEDs were already connected to the Master server, they rejected the Standby connections. This caused the driver thread to flood with retries, blocking variable updates entirely.

    Furthermore, when we completely delete the historian files, the Runtimes run completely fine without any problems. The issue only resurfaces when we put the archive files back into their original directory. By the way, the archive recording cycle for analog data is set to 3 seconds.

    Any guidance from the community or COPA-DATA engineers would be greatly appreciated.

      Disclaimer

      This document governs the use of our Community Forum. By registering and using the platform, you accept these conditions.

      The COPA-DATA Community Forum serves to encourage the exchange of information and experience about the zenon software between forum users respectively zenon users.

      Please mind that any published information on the Community Forum is the subjective opinion and view based on the experience and the level of knowledge of the author. COPA-DATA does not overtake any responsibility for the content and the accuracy of the shared information.

      Users of the Community Forum are encouraged to share only well-founded experiences and to point out any risks associated with the implementation of proposed solutions to problems. COPA-DATA at its absolute discretion, reserves the right to moderate the forum. In this connection COPA-DATA may remove any information containing false facts, potentially dangerous solutions, bad language or content that may insult, degrade or discriminate others. COPA-DATA may block a non-complying user from forum access if the user violated this provision.

      COPA-DATA reserves the right to change this document from time to time at own discretion.


      Ing. Punzenberger COPA-DATA GmbH
      Karolingerstraße 7b · 5020 Salzburg · Austria
      www.copadata.com