Zenon 8.20 Redundancy — Standby Server IEC 61850 Driver TCP Connect Errors Causing Variable Freeze

Zenon 8.20 Redundancy — Standby Server IEC 61850 Driver TCP Connect Errors Causing Variable Freeze

  • Zenon Energy Edition 8.20
  • Software Redundancy (Master / Standby)
  • IEC 61850 driver, 50+ IEDs
  • Windows Server 2019, 32 GB RAM, Xeon Silver 4208

    We are running a dual-server redundancy setup. Afte the Standby server started freezing all variable values shortly after startup.

    Symptoms observed:

    • On first startup, Standby server receives values correctly
    • After historian sync completes, all variable values disappear or freeze
    • Some variables partially visible (e.g. Va visible, Vb/Vc missing)
    • Opening trend or alarm pages on Standby triggers another historian sync and causes values to freeze again
    • Both servers show Master in system variables, Standby field always empty — classic split-brain behavior
    • Switching active server restores values on the previously frozen server
    • System restart takes approximately 15 minutes due to large archive size (~40 GB .arx folder, ~1 million files)

    Diagnosis Viewer findings:

    • Hundreds of OSI4 / T5BusExlec850m errors
    • TCP connect errors to all IED IP addresses simultaneously from Standby server
    • IEC 61850 R2 connection errors continuously repeating
    • 529 error lines, queue nearly overflowing

    • Initial assumption was resource pressure due to large historian archive. We renamed the 40 GB .arx folder, recompiled and redeployed the project — system worked correctly on both servers.

    We then attempted to migrate the last 3 months of historian data (customer requirement) in 3 batches. After the 3rd batch was copied, Standby server froze again.

    RAM usage was checked during freeze — only 44% utilized (approximately 14 GB), so memory pressure was ruled out.

    Diagnosis Viewer clearly showed the Standby server IEC 61850 driver attempting TCP connections to all IEDs simultaneously, even while in Standby mode. Since all IEDs were already connected to the Master server, they rejected the Standby connections. This caused the driver thread to flood with retries, blocking variable updates entirely.

    Furthermore, when we completely delete the historian files, the Runtimes run completely fine without any problems. The issue only resurfaces when we put the archive files back into their original directory. By the way, the archive recording cycle for analog data is set to 3 seconds.

    Any guidance from the community or COPA-DATA engineers would be greatly appreciated.