RTMediaEngineSRV core dumps on Linux Mint 19.3 (Ubuntu 18.04 bionic)

Michael Cox1709161710 · February 9, 2020

I used 2.2 of citrix-hdx-realtime-media-engine on Linux mint 18.* for years logging on to work and using skype for business without issue.

However on linux mint 19.3 (based on ubuntu 18.04) it core dumps at some point into a skype call and I am disconnected from the call.

It does this for versions 2.2, 2.4 and 2.8 though I would like to use 2.4 has it matches the remote destination.

It looks like its trying to output some log line - maybe if I could turn off RTMediaEngineSRV debug logging I might be able to get it to stay up.

Has anyone seen this please and knows how to fix?

Feb  8 21:04:43 antibes kernel: [71151.187132] traps: RTMediaEngineSR[3596] general protection fault ip:7fc967c71e33 sp:7fc960823840 error:0 in libc-2.27.so[7fc967b98000+1e7000]

Feb 8 21:04:44 antibes systemd-coredump[4719]: Process 3588 (RTMediaEngineSR) of user 1000 dumped core.#012#012Stack trace of thread 3596:



#012#0 0x00007fc967c71e33 __strftime_internal (libc.so.6)

#012#1 0x00007fc967c74266 __GI___strftime_l (libc.so.6)

#012#2 0x000000000118e2b5 _ZN3vos3log11BasicLayout6FormatERKNS0_5EventE (RTMediaEngineSRV)

#012#3 0x000000000117db34 _ZNK3vos3log14LayoutAppender8DoLayoutERKNS0_5EventE (RTMediaEngineSRV)

#012#4 0x000000000117dc27 _ZN3vos3log14MemoryAppender8DoAppendERKNS0_5EventE (RTMediaEngineSRV)

#012#5 0x00000000011822fa _ZN3vos3log8Appender6AppendERKNS0_5EventE (RTMediaEngineSRV)

#012#6 0x000000000118ae9c _ZN3vos3log8Category13CallAppendersERKNS0_5EventE (RTMediaEngineSRV)

#012#7 0x000000000118ae3e _ZN3vos3log8Category13CallAppendersERKNS0_5EventE (RTMediaEngineSRV)

#012#8 0x000000000118af5e _ZN3vos3log8Category9LogAlwaysERKNS0_8PriorityEPKcP13__va_list_tag (RTMediaEngineSRV)

#012#9 0x000000000118b684 _ZN3vos3log8Category5DebugEPKcz (RTMediaEngineSRV)

#012#10 0x0000000001170c92 n/a (RTMediaEngineSRV)#012#11 0x0000030000080008 n/a (n/a)

Edited February 9, 2020 by michaeldjcox
Found additional log output that might be meaningful

Daniel Szomboti · February 10, 2020

23 hours ago, Michael Cox1709161710 said:
I used 2.2 of citrix-hdx-realtime-media-engine on Linux mint 18.* for years logging on to work and using skype for business without issue.

However on linux mint 19.3 (based on ubuntu 18.04) it core dumps at some point into a skype call and I am disconnected from the call.

It does this for versions 2.2, 2.4 and 2.8 though I would like to use 2.4 has it matches the remote destination.

It looks like its trying to output some log line - maybe if I could turn off RTMediaEngineSRV debug logging I might be able to get it to stay up.

Has anyone seen this please and knows how to fix?
Feb  8 21:04:43 antibes kernel: [71151.187132] traps: RTMediaEngineSR[3596] general protection fault ip:7fc967c71e33 sp:7fc960823840 error:0 in libc-2.27.so[7fc967b98000+1e7000]

Feb 8 21:04:44 antibes systemd-coredump[4719]: Process 3588 (RTMediaEngineSR) of user 1000 dumped core.#012#012Stack trace of thread 3596:



#012#0 0x00007fc967c71e33 __strftime_internal (libc.so.6)

#012#1 0x00007fc967c74266 __GI___strftime_l (libc.so.6)

#012#2 0x000000000118e2b5 _ZN3vos3log11BasicLayout6FormatERKNS0_5EventE (RTMediaEngineSRV)

#012#3 0x000000000117db34 _ZNK3vos3log14LayoutAppender8DoLayoutERKNS0_5EventE (RTMediaEngineSRV)

#012#4 0x000000000117dc27 _ZN3vos3log14MemoryAppender8DoAppendERKNS0_5EventE (RTMediaEngineSRV)

#012#5 0x00000000011822fa _ZN3vos3log8Appender6AppendERKNS0_5EventE (RTMediaEngineSRV)

#012#6 0x000000000118ae9c _ZN3vos3log8Category13CallAppendersERKNS0_5EventE (RTMediaEngineSRV)

#012#7 0x000000000118ae3e _ZN3vos3log8Category13CallAppendersERKNS0_5EventE (RTMediaEngineSRV)

#012#8 0x000000000118af5e _ZN3vos3log8Category9LogAlwaysERKNS0_8PriorityEPKcP13__va_list_tag (RTMediaEngineSRV)

#012#9 0x000000000118b684 _ZN3vos3log8Category5DebugEPKcz (RTMediaEngineSRV)

#012#10 0x0000000001170c92 n/a (RTMediaEngineSRV)#012#11 0x0000030000080008 n/a (n/a)

Hi Michael.

We have documented the method to turn off logging in this article: https://docs.citrix.com/en-us/hdx-optimization/2-4-ltsr/configuration.html (bottom of page)

Any trouble ping me.

Daniel

Michael Cox1709161710 · February 10, 2020

Hi

Thanks for responding.

I can only see references to changes in windows registry in that document

The client system running RTMediaEngineSRV is linux.

I guess there must be some other place configuration changes can be done.

There is various config in /etc/icaclient/config and /opt/Citrix/ICAClient/ but I cannot see anything that seems to match the windows registry.

The command line flags used to run RTMediaEngineSRV seem to be embedded in a binary file called HDXRTME.so

michael@nice:/opt/Citrix/ICAClient$ fgrep -Hr "rtc_debug" *
michael@nice:/opt/Citrix/ICAClient$ ps wwwaux | grep RTM
michael  12951  0.7  0.4 2395148 40060 ?       Sl   08:34   1:00 RTMediaEngineSRV --port:7102 --requested_version:2.4 --supported_versions:2.3,2.2,2.1,2.0 --disable_rtc_logging:0 --disable_rtme_logging:0
michael  15874  0.0  0.0  14424  1004 pts/0    S+   10:54   0:00 grep --color=auto RTM
michael@nice:/opt/Citrix/ICAClient$ 
michael@nice:/opt/Citrix/ICAClient$ 
michael@nice:/opt/Citrix/ICAClient$ fgrep -Hr "disable_rtc" *
Binary file HDXRTME.so matches
Binary file rtme/RTMediaEngineSRV matches

Daniel Szomboti · February 10, 2020

2 minutes ago, Michael Cox1709161710 said:
Hi

Thanks for responding.

I can only see references to changes in windows registry in that document

The client system running RTMediaEngineSRV is linux.

I guess there must be some other place configuration changes can be done.

There is various config in /etc/icaclient/config and /opt/Citrix/ICAClient/ but I cannot see anything that seems to match the windows registry.

The command line flags used to run RTMediaEngineSRV seem to be embedded in a binary file called HDXRTME.so
michael@nice:/opt/Citrix/ICAClient$ fgrep -Hr "rtc_debug" *
michael@nice:/opt/Citrix/ICAClient$ ps wwwaux | grep RTM
michael  12951  0.7  0.4 2395148 40060 ?       Sl   08:34   1:00 RTMediaEngineSRV --port:7102 --requested_version:2.4 --supported_versions:2.3,2.2,2.1,2.0 --disable_rtc_logging:0 --disable_rtme_logging:0
michael  15874  0.0  0.0  14424  1004 pts/0    S+   10:54   0:00 grep --color=auto RTM
michael@nice:/opt/Citrix/ICAClient$ 
michael@nice:/opt/Citrix/ICAClient$ 
michael@nice:/opt/Citrix/ICAClient$ fgrep -Hr "disable_rtc" *
Binary file HDXRTME.so matches
Binary file rtme/RTMediaEngineSRV matches

If you set the keys on the VDA side, that will pass down to the media engine on linux and tell it to not use logging. No need for local config.

Daniel

Michael Cox1709161710 · February 10, 2020

So I need to set registry keys on the windows box at the remote end?

That will be tricky. Big company. Don't think they will let me tweak any registry settings outside of their normal PC setup even if I asked them :-(

Daniel Szomboti · February 10, 2020

Well the keys are HKCU, don't see a reason they would be mad about it :)

If you can get me a memory dump and the logs it generated to that point and open a support ticket to my attention, I can certainly look into the crash.

Regards,

Daniel

Michael Cox1709161710 · February 10, 2020

I had a go with reg edit and i could do the first two but accessed denied on the last one.

I will see about getting a core dump and logs to you.

Michael Cox1709161710 · February 11, 2020

Sent them in a PM.

The crashes seem to occur every 20 mins plus or minus a minute but I cannot see anything obvious in the media engine logs.

Michael Cox1709161710 · February 18, 2020

Its fixed - or at least I have a workaround

.

Daniel noticed that the MediaEngine logs contained a lot of this

MediaEngineService[25774]: Feb 17 21:28:53.059 [NetworkMonitorThread] NetworkMonitorThread debug: IP Address table changed: RTM_NEWADDR.
MediaEngineService[25774]: Feb 17 21:28:53.059 [NetworkMonitorThread] NetworkMonitorThread debug: IP Address table changed: RTM_NEWADDR.
MediaEngineService[25774]: Feb 17 21:28:53.059 [NetworkMonitorThread] NetworkMonitorThread debug: IP Address table changed: RTM_NEWADDR.

I found using "ip monitor all" that this coincided on each instance with this:

michael@antibes:~$ ip monitor all
[ADDR]2: enp5s0    inet6 2a00:23c6:6288:7001:eba:dca0:359c:90a5/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 300sec preferred_lft 120sec
[ADDR]2: enp5s0    inet6 2a00:23c6:6288:7001:69c2:b693:aa9e:a0cd/64 scope global temporary dynamic 
       valid_lft 300sec preferred_lft 120sec
[ADDR]2: enp5s0    inet6 fe80::9d6b:a353:324f:950c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

I disabled ipV6 in GRUB and the core dumps in syslog disappeared.

There is obviously more going on here and many questions unanswered:

Why is there a ip6 network address change every 30 seconds.
Why does that cause RTMediaEngineSRV to crash every 20 minutes because of those changes?
Why does it crash trying to format a time for logging?

But I have a way forward now

Maxim Samo · May 8, 2020

OMG... Yesterday morning I have found this problem as well on my Ubuntu 19.10. It seems that RTMediaEngineSRV crashes whenever there's an IPv6 network change. Privacy extensions will cause such behavior as documented in RFC 4941. Those can be disabled setting net.ipv6.conf.all.use_tempaddr = 0 and net.ipv6.conf.default.use_tempaddr = 0 however even then I had disconnects every 15-20 minutes as it seems that ipv6 does "something" that then triggers the RTM_NEWADDR error.

Completely disabling ipv6 on the client machine works but I believe that there's a bug in RTMediaEngineSRV that is causing this crash.

Daniel Szomboti · May 8, 2020

25 minutes ago, Maxim Samo said:

OMG... Yesterday morning I have found this problem as well on my Ubuntu 19.10. It seems that RTMediaEngineSRV crashes whenever there's an IPv6 network change. Privacy extensions will cause such behavior as documented in RFC 4941. Those can be disabled setting net.ipv6.conf.all.use_tempaddr = 0 and net.ipv6.conf.default.use_tempaddr = 0 however even then I had disconnects every 15-20 minutes as it seems that ipv6 does "something" that then triggers the RTM_NEWADDR error.

Completely disabling ipv6 on the client machine works but I believe that there's a bug in RTMediaEngineSRV that is causing this crash.

Yes, that's a bug RTOP doesn't support IPv6. We are aware of it and will be taken care of.

Regards,
Daniel

Maxim Samo · May 8, 2020

3 hours ago, Daniel Szomboti said:

Yes, that's a bug RTOP doesn't support IPv6. We are aware of it and will be taken care of.

Regards,
Daniel

Thanks... is that on all platforms or just linux?

Daniel Szomboti · May 8, 2020

1 hour ago, Maxim Samo said:

Thanks... is that on all platforms or just linux?

Just linux luckily :)

Daniel Venema · September 15, 2020

Same use case for me and same issue. Ubuntu 20.04 with "HDX_RealTime_Media_Engine_2.9_for_Linux_x64".

I just did the regedit trick for the two that I was allowed to do (the ones in HKEY_CURRENT_USER). Maybe that's enough to stop the crashes for now. Hope it will be fixed in the next release (2.10 ?).

xxxxx@xxxxxx:/var/crash$ head _opt_Citrix_ICAClient_rtme_RTMediaEngineSRV.1000.crash -n15
ProblemType: Crash
Architecture: amd64
CrashCounter: 1
CurrentDesktop: ubuntu:GNOME
Date: Tue Sep 15 10:31:28 2020
DistroRelease: Ubuntu 20.04
ExecutablePath: /opt/Citrix/ICAClient/rtme/RTMediaEngineSRV
ExecutableTimestamp: 1582396480
ProcCmdline: RTMediaEngineSRV --port:7102 --requested_version:2.8 --supported_versions:2.10,2.9,2.8,2.7,2.6,2.5,2.4,2.3,2.2 --disable_rtc_logging:0 --disable_rtme_logging:0
ProcCwd: /home/xxxxx
ProcEnviron:
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=nl_NL.UTF-8
SHELL=/bin/bash

Edited September 15, 2020 by dvenema
added line about doing the regedit trick

Daniel Venema · December 16, 2020

Looks like with the release of "HDX_RealTime_Media_Engine_2.9.200_for_Linux" the problem has been fixed. After upgrading today, I experienced no more sudden Skype call terminations.

Thorsten Niehues · January 26, 2022

On 12/16/2020 at 8:15 PM, Daniel Venema said:

Looks like with the release of "HDX_RealTime_Media_Engine_2.9.200_for_Linux" the problem has been fixed. After upgrading today, I experienced no more sudden Skype call terminations.

Where did you get this 2.9.200 Version?

On the download site https://www.citrix.com/downloads/citrix-receiver/additional-client-software/hdx-realtime-media-engine-latest.html

I downloaded the "latest Version" from Mar 6, 2020 - It only says 2.9

But when extracted I see the file citrix-hdx-realtime-media-engine_2.9.0-2330_amd64.deb which is already installed:

apt list |grep -i hdx
citrix-hdx-realtime-media-engine/now 2.9.0-2330 amd64 [installed,local]

Unfortunately I still get these random disconnects:

tail -2 /var/log/kern.log
Jan 26 08:20:45 myPcName kernel: [ 1300.799837] traps: RTMediaEngineSR[5526] general protection fault ip:7fbf563bc33a sp:7fbf4e1f77a0 error:0 in libc-2.31.so[7fbf562e1000+1e7000]
Jan 26 08:39:54 myPcName kernel: [ 2450.017392] traps: RTMediaEngineSR[8463] general protection fault ip:7fd8fca9f33a sp:7fd8f88df7a0 error:0 in libc-2.31.so[7fd8fc9c4000+1e7000]

PS the random disconnects started after installing Citrix 20.12 and some libraries to make Microsoft Teams working... for details see https://discussions.citrix.com/topic/415288-linux-host-teams-on-windows-client-does-not-detect-microphone/

Thorsten Niehues · January 31, 2022

On 2/10/2020 at 11:58 AM, Daniel Szomboti said:

If you set the keys on the VDA side, that will pass down to the media engine on linux and tell it to not use logging. No need for local config.

Daniel

I tried the registry settings disable logging - but the error persists.. any more suggestions?

Thorsten Niehues · January 31, 2022

On 1/26/2022 at 8:53 AM, Thorsten Niehues said:
Where did you get this 2.9.200 Version?

On the download site https://www.citrix.com/downloads/citrix-receiver/additional-client-software/hdx-realtime-media-engine-latest.html

I downloaded the "latest Version" from Mar 6, 2020 - It only says 2.9

But when extracted I see the file citrix-hdx-realtime-media-engine_2.9.0-2330_amd64.deb which is already installed:
apt list |grep -i hdx
citrix-hdx-realtime-media-engine/now 2.9.0-2330 amd64 [installed,local]
Unfortunately I still get these random disconnects:
tail -2 /var/log/kern.log
Jan 26 08:20:45 myPcName kernel: [ 1300.799837] traps: RTMediaEngineSR[5526] general protection fault ip:7fbf563bc33a sp:7fbf4e1f77a0 error:0 in libc-2.31.so[7fbf562e1000+1e7000]
Jan 26 08:39:54 myPcName kernel: [ 2450.017392] traps: RTMediaEngineSR[8463] general protection fault ip:7fd8fca9f33a sp:7fd8f88df7a0 error:0 in libc-2.31.so[7fd8fc9c4000+1e7000]
PS the random disconnects started after installing Citrix 20.12 and some libraries to make Microsoft Teams working... for details see https://discussions.citrix.com/topic/415288-linux-host-teams-on-windows-client-does-not-detect-microphone/

ATTENTION as of Jan 31, 2022 the Citrix site for latest Version

https://www.citrix.com/downloads/citrix-receiver/additional-client-software/hdx-realtime-media-engine-latest.html

still holds the out-dated version 2.9.0 use Version 2.9.4 from

https://www.citrix.com/de-de/downloads/citrix-receiver/additional-client-software/hdx-realtime-media-engine-29400.html

Also answered here : https://askubuntu.com/q/1390270/230083

Edited January 31, 2022 by Thorsten Niehues
added link to askUbuntu for faster problem solving

RTMediaEngineSRV core dumps on Linux Mint 19.3 (Ubuntu 18.04 bionic)

Question

Link to comment

17 answers to this question

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Create an account or sign in to comment

Create an account

Sign in