IGEL OS Thin Clients + MS Teams Optimization - Duplicate camera detection, Video drops from call after exactly 3-4 seconds

[Alex Bennett]

We are having significant issues with using MS Teams optimization on IGEL OS Thin Clients. The main issue we're seeing is that a webcam is detected twice. Video previews will work just fine, and you can start a video call, but after exactly 3-4 seconds into a call, the video dies and Teams says it can't contact the camera anymore. Then video previews are also broken until you get a fresh VDI session.

 

After discussing the issue in IGEL's Slack, they suggested that this was a known issue on Microsoft's end, and they were working with Citrix to resolve. Can we get some confirmation if this is the case? I checked the live article but hadn't seen any updates about this issue. We're also finding some buggy behavior with keeping non-video calls connected, though I think we found a workaround for that: https://discussions.citrix.com/topic/410930-teams-calls-in-citrix-often-not-connecting/?do=findComment&comment=2072990

[Daniel Szomboti]

15 hours ago, Alex Bennett said:

We are having significant issues with using MS Teams optimization on IGEL OS Thin Clients. The main issue we're seeing is that a webcam is detected twice. Video previews will work just fine, and you can start a video call, but after exactly 3-4 seconds into a call, the video dies and Teams says it can't contact the camera anymore. Then video previews are also broken until you get a fresh VDI session.

 

After discussing the issue in IGEL's Slack, they suggested that this was a known issue on Microsoft's end, and they were working with Citrix to resolve. Can we get some confirmation if this is the case? I checked the live article but hadn't seen any updates about this issue. We're also finding some buggy behavior with keeping non-video calls connected, though I think we found a workaround for that: https://discussions.citrix.com/topic/410930-teams-calls-in-citrix-often-not-connecting/?do=findComment&comment=2072990

Do you have the bug reference by any chance?

[Stefan Schober]

Hello

 

we have the same Problem. It seems like that the HdxRtcEngine process on the Client will be terminated after you start a call in Teams in your VDi Session. And so no HDX Optimation will not work anymore and the Video is gone.

 

It can be replicated in Ubuntu OS or Elux OS, with the new Workspace app 2010 for Linux

[Daniel Szomboti]

43 minutes ago, Stefan Schober said:

Hello

 

we have the same Problem. It seems like that the HdxRtcEngine process on the Client will be terminated after you start a call in Teams in your VDi Session. And so no HDX Optimation will not work anymore and the Video is gone.

 

It can be replicated in Ubuntu OS or Elux OS, with the new Workspace app 2010 for Linux

Can you capture a memory dump and open a TS case please? 

Daniel 

[Stefan Schober]

Capture the whole Memory or the Process Memory itself? Only when den Process will be terminated?

 

I try procmem on Ubuntu, but it will only Dump the Process when its running. not when its closing.

[Alex Bennett]

On 11/6/2020 at 6:16 AM, Daniel Szomboti said:

Do you have the bug reference by any chance?

 I don't unfortunately, I've just heard rumors but haven't found anything to back it up yet. The most detail I could get was that there was an update installed to O365 tenants around 10/15 that may be related to the issue, and that someone was contacting Microsoft support to get the Teams shim part of it downgraded. I'm looking for info on it.

[Daniel Szomboti]

1 hour ago, Alex Bennett said:

 I don't unfortunately, I've just heard rumors but haven't found anything to back it up yet. The most detail I could get was that there was an update installed to O365 tenants around 10/15 that may be related to the issue, and that someone was contacting Microsoft support to get the Teams shim part of it downgraded. I'm looking for info on it.

I see. If you cna capture a memory dump I'd be interested. Open a ticket if you do and give me the ticket number in an IM. 

[Alex Bennett]

6 minutes ago, Daniel Szomboti said:

I see. If you cna capture a memory dump I'd be interested. Open a ticket if you do and give me the ticket number in an IM. 

Sounds good, I'll try to do that. I also opened a ticket with Microsoft and will let you know what I hear from them.

[Mike_B]

We see the same issue on Wyse clients running ThinOS, and also on 10Zig running their minimal-OS equivalent.   Only with Teams Meetings though - person to person calls tend to behave themselves, at least until you try a Teams Meeting.   The behaviour with Meetings is significantly worse.

 

All our terminals run the Workspace for Linux application to connect, and all behave in much the same way.  I have heard from both Dell and 10Zig independently that MS made a change to the Meetings backend a few weeks ago and it's got progressively worse since then, the message from both companies about something breaking on the endpoint that then stops HDX Optimization working until it's all reset.   The timing also ties in perfectly with our experience when it basically worked OK until about 16th October, and now Teams meetings are horribly unreliable from any Linux endpoint.   Basically, they just don't work any more.

 

I have a ticket with Citrix about it - the support engineer asked me for our TenantID, to be dug out from a Teams logfile.   Nothing else, just that - which suggests heavily they know exactly what caused it and are working with MS about it, using our TenantID as a reference point to check settings etc.

 

So no, you're not alone, it's gone badly wrong in the last 2-3 weeks.

Edited November 10, 2020 by mburnst412
Clarification

[Alex Bennett]

After discussing with IGEL some more, they were able to give me a private build of the latest firmware that seems to fix the issue, we are testing that now. From what they told me, the fix actually came from Citrix via an updated Workspace (though in the firmware, it still shows the version as 20.10). So it seems like there will be an update to Linux Workspace that will contain the fix, and then that should filter downstream to thin client firmwares.

[Mike_B]

Well that is good news - kudos to IGEL for sorting that so quickly, and hopefully they'll release to production if testing is successful.

 

Hopefully the other terminal manufacturers will get a similar fix out for private testing too, if it's a Workspace update.   

[Mike_B]

QQ - do you still see multiple entries for the webcam after the private firmware build is applied?   I get that too - although whichever one is selected seems to make no difference.

[Alex Bennett]

5 minutes ago, Michael Burnstead1709159565 said:

QQ - do you still see multiple entries for the webcam after the private firmware build is applied?   I get that too - although whichever one is selected seems to make no difference.

I do, I mentioned that to IGEL but they said they've always seen it like that from when they first added it. So maybe that wasn't the cause of the issue originally, but seemed strange.  Though as long as both of them work it's not a big deal.

[Mike_B]

OK thanks, I see the same on all of my Dell/Wyse, the 10Zig and my IGEL units.   Just a quirk I suspect (maybe from different possible resolutions) but I do get the odd user questioning it.

[Mike_B]

Well here's a turnup for the books - it's fixed!

 

Root cause was a shim package, written by Citrix but distributed by MS into VDIs when Teams connects to the tenant in O365.   On 15th October, version 1.7 was rolled out and all us lot on Linux based terminals started having major problems.   Today, it was globally rolled back to version 1.3.0.5 and now all three of my terminal types are suddenly working again...

[Guoyi Zhou]

abennet240, could you please supply more background for this issue? such as configuration info( iGEL OS version, Teams version, Linux CWA version, etc) and the scenario you are running on,   we will try to reproduce this problem in lab.  thanks. 

[Alex Bennett]

@Guoyi Zhou Sure, this is the setup we are using:

 

CWA: 1912 CU1

IGEL OS: 11.04.200+

Teams Version: 1.3.00.26064 (x64)

However to be able to reproduce it you need the shim version 1.7 in your O365 tenant, as @Michael Burnstead1709159565 mentioned. From this thread we can see though that it seems to affect any Linux Workspace app that's using that shim version.

Speaking of the shim, if I understand correctly this is something made by Citrix that Microsoft ships to O365 tenants? If so, was there any support bulletin about this? I see on the live article that it mentions when it was rolled back, but for an outage of this magnitude I would expect that we should have more detail from Citrix, as well as maybe a service notification from Microsoft. The only way I found out about the issue with the shim was rumors on support forums, after spending a great deal of time troubleshooting.

 

I get that mistakes happen and know there are lots of moving parts to make Teams work, but I think Citrix and Microsoft need to be more transparent in these situations.

[Guoyi Zhou]

@Alex Bennett  thanks for the reply,   I will forward your concern to our Product Manager. 

 

I still have two questions for you:

1. The version you supplied should be VDA version but not CWA version. could you please supply the version of CWA?

2. You mentioned "Duplicate camera detection", do you mean there are duplicated entries in the list of menu "Settings --> Devices --> Camera" for one physical camera? 

 

CC @Liang Chen1709157604 for awareness. 

 

 

 

[Mike_B]

I can't speak for 'abennet240' but on my environment, yes the camera appears more then once in the dropdown list.

 

I am running Workspace 2006 - those of us with thin clients are much more reliant on vendors to get updates, and also we can't make any changes at all to it's settings beyond what is exposed on their terminal management platforms.   Currently, Workspace 2006 is the latest version I am able to put on my Wyse 5070 devices.