Printing issue with session freeze since R04 on Windows 2008 R2 SP1

KEN SANDHU1709154392 · July 31, 2014

Here is our environment:

- 13 Xenapp 6.5 Servers with Rollup Pack 4

- Universal Print Driver

- Windows 2008 R2 SP1

- Receiver 3.4.300.10 Enterprise

- Windows 7 64-bit clients

- Web Interface 5.4

- Secure Gateway 3.3

The Issue:

When a user goes to print from any application (Excel, Word, Outlook, Reader, etc...) the print dialog box freezes their session. The dialog never shows the printers for the session and the window is movable while no other interaction with the session is allowed. Even waiting 10-15 minutes the session is still frozen and no print dialog is shown.

Another item we saw related is that when you RDP onto any of our XenApp boxes and go to "Device and Printers", the system sits there waiting to enumerate the printers and devices and stays there until the band-aid solution is done (below).

The Band-aid we figured out:

We restart the Print Spooler service which also restarts the Citrix Print Manager service and that instantly unfreezes the effected user sessions. One thing we did note is that the service hangs on stopping Citrix Print Manager service and we have to manually kill the cpsvc.exe process in order to bring it back up.

For now, we've created scripts to kill the cpsvc.exe and restart the Print Spooler and Citrix Print Manager services.

Citrix Case Support:

Citrix Case # 63566087

Initial recommendation from the engineer was to set our Citrix Print Manager service to run as "Local System" rather than "Local Service" account. We made that change on all boxes last night and no change in freezes this morning.

Next step is to try and get a server with this issue and collect the CDF logs to find root cause.

Anyone having similar issues? Any private fixes available that we should ask for?

KEN SANDHU1709154392 · August 13, 2014

After providing a CDF trace, cpsvc.exe dump, and a fully memory dump of a Citrix box to Citrix for analysis, it turns out that we had a print driver from Kyocera that was the culprit. This issue arose after R04 because prior to R04, we didnt' have this issue with this driver.

The root to that issue was that our Citrix Policy to not install drivers wasn't working as expected. We kept the servers clean by using the Print Detective utility to remove drivers and then restart "Print Spooler" multiple times a day so we could test the theory of print drivers and it definitely improved the stability of the system.

We had to implement http://support.citrix.com/article/CTX128786 in our Group Policy for our XenApp farm to stop having them get installed automatically. So far, we're on day 2 without ANY issues relating to Citrix printing so only time will tell.

Craig Moglia · July 31, 2014

Since going to R04 I have had alot of printer start going offline without a fix this far.

I tried resetting the Print Spooler, needed to end task on CPSVC since that service hung trying to shutdown.

We use the Citrix Univeral Print driver for all our printers, but still some but not others go offline.

Not sure if this is directly related to your issue. I have not seen any freezing.

IT Department1709151537 · October 11, 2014

We are having this exact same problem. It just started out of the blue with no changes to the environment. Our band-aid is the same as yours for right now. Citrix support is providing much support. No Kyocera drivers in our environment. I made the change in the article linked above. Will see if that helps.

Colby Vrbka1709152508 · December 17, 2014

I am having the same issue. I have no 3rd party print drivers install on the Citrix Servers. I tested today and when it happened I restarted the Citrix Print Manager and that did not resolve the issue. When I restarted Print Spooler that did fix the issue. Did you guys get a hotfix or anything to fix the issue?

Jef Bogaerts · December 18, 2014

Did you update to 7.6? We have the same problem on our XenApp 6.5 HRP 4 farm, which started since we upgraded from UPS 1.1 to 7.6.

We have 2 problems, which seem related:

- when printing from Word, Outlook... the dialog box hangs, same problem as the OP.

- logon scripts hang because printers are being mapped via these scripts and the spooler process doesn't allow them to be mapped as it seems to be hanging. The devices and printers windows also hangs, no printers are being showed.

We need to restart the print spooler on the XenApp to continue. The scripts start running after that and everything is ok.

I also noticed that on some printservers (we have 12), the UPServer process used 800MB memory, which is a lot (normally about 15MB).

We really don't want to rollback, as the update has solved a number of other printing problems. If we can't find the solution, we will be opening a support case.

We did some investigation ourselves:

Printspooler hangs with hundreds of threads waiting.

Spooler-memory dump analysis :

0:000> !analyze -v -hang

*******************************************************************************

* *

* Exception Analysis *

* *

*******************************************************************************

GetPageUrlData failed, server returned HTTP status 404

URL requested: http://watson.microsoft.com/00000000.htm?Retriage=1

FAULTING_IP:

+882b270

00000000`00000000 ?? ???

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)

ExceptionAddress: 0000000000000000

ExceptionCode: 80000003 (Break instruction exception)

ExceptionFlags: 00000000

NumberParameters: 0

FAULTING_THREAD: 000000000000001c

BUGCHECK_STR: HANG

PROCESS_NAME: spoolsv.exe

ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>

EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

DERIVED_WAIT_CHAIN:

Dl Eid Cid WaitType

-- --- ------- --------------------------

0 6558.65c8 Speculated (Triage) -->

x 12 6558.878 Critical Section -->

x 28 6558.841c Critical Section --^

WAIT_CHAIN_COMMAND: ~0s;k;;~12s;k;;~28s;k;;

BLOCKING_THREAD: 000000000000841c

DEFAULT_BUCKET_ID: APPLICATION_HANG_DEADLOCK

PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_DEADLOCK

LAST_CONTROL_TRANSFER: from 00000000775fd7a8 to 00000000776005fa

STACK_TEXT:

00000000`065ae5f8 00000000`775fd7a8 : 00000000`065ae738 00000000`00001e9c 00000000`00000018 000007fe`de85ef2e : ntdll!ZwWaitForSingleObject+0xa

00000000`065ae600 00000000`775fd69b : 00000000`00000000 00000000`065aed48 00000000`00000000 00000000`04db3fe8 : ntdll!RtlpWaitOnCriticalSection+0xe8

00000000`065ae6b0 000007fe`de8a9c84 : 00000000`065aee20 00000000`04ba1620 00000000`041248d8 000007fe`de7b8698 : ntdll!RtlEnterCriticalSection+0xd1

00000000`065ae6e0 000007fe`de7fd618 : 00000000`00000000 00000000`065aed48 00000000`041248d8 00000000`04ba1620 : upprov!InitializePrintProvidor+0xb87a4

00000000`065ae710 000007fe`de8e3d4a : 00000000`065ae770 00000000`04db3fe0 00000000`00000000 00000000`00001414 : upprov!InitializePrintProvidor+0xc138

00000000`065ae740 000007fe`de8e4454 : 00000000`04db3fa0 00000000`065ae8d0 00000000`00000000 00000000`00000000 : upprov!InitializePrintProvidor+0xf286a

00000000`065ae8b0 000007fe`de8c5bf7 : 00000000`04db3fa0 00000000`057df230 00000000`00000008 00000000`065aee20 : upprov!InitializePrintProvidor+0xf2f74

00000000`065ae900 000007fe`de80e5d6 : 00000000`04d551e0 000007fe`de939e00 00000000`04d59a10 00000000`041248d8 : upprov!InitializePrintProvidor+0xd4717

00000000`065ae9c0 000007fe`de839800 : 000007fe`de939d80 00000000`041248d8 00000000`065aeb98 00000000`065aee20 : upprov!InitializePrintProvidor+0x1d0f6

00000000`065aeb10 000007fe`de838cec : 000007fe`de939ef0 00000000`041248d8 00000000`065aeb98 00000000`065aee20 : upprov!InitializePrintProvidor+0x48320

00000000`065aeb60 000007fe`de7f1909 : 000007fe`de939ef0 00000000`041248d8 00000000`065aed48 00000000`065aee20 : upprov!InitializePrintProvidor+0x4780c

00000000`065aec10 00000000`fffc1abd : 00000000`041248d8 00000000`065aed48 00000000`065aee20 00000000`00000000 : upprov!InitializePrintProvidor+0x429

00000000`065aecc0 00000000`fffc809f : 00000000`041248d8 00000000`041248d8 00000000`05d3ad60 00000000`04ba1620 : spoolsv!TryOpenPrinterAndCache+0x63

00000000`065aed00 00000000`fffb264d : 00000000`0000007b 00000000`041a0e70 00000000`04bb7780 00000000`05d3adc0 : spoolsv!RouterOpenPrinterW+0x2f3

00000000`065aedb0 00000000`fffb25b5 : 00000000`04bb7780 00000001`000002ea 00000000`065af4c0 000007fe`fe2f61ff : spoolsv!OpenPrinterExW+0x49

00000000`065aedf0 00000000`fffb24d6 : 00000000`00000000 00000000`041a0e70 00000000`00000000 00000000`041248d8 : spoolsv!YOpenPrinterEx+0xc4

00000000`065aee50 000007fe`fe2dff85 : 00000000`00000006 00000000`065aeee0 00000000`065af320 00000000`065aeee0 : spoolsv!RpcSplOpenPrinter+0x9c

00000000`065aeeb0 000007fe`fe2d4de2 : 00000000`065af2f0 00000001`000002c2 00000000`00000040 00000001`00005258 : rpcrt4!Invoke+0x65

00000000`065aef20 000007fe`fe2d17bd : 00000000`0418a2c0 00000000`00000000 00000000`0548fb70 00000000`004a1f20 : rpcrt4!NdrStubCall2+0x32a

00000000`065af540 000007fe`fe2d3254 : 00000000`0000001d 00000001`00000001 00000000`00000002 00000000`00000000 : rpcrt4!NdrServerCall2+0x1d

00000000`065af570 000007fe`fe2d33b6 : 00000000`00000000 00000000`00000000 00000000`065af720 00000000`00000000 : rpcrt4!DispatchToStubInCNoAvrf+0x14

00000000`065af5a0 000007fe`fe2d3aa9 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0548fa20 : rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0x146

00000000`065af6c0 000007fe`fe2d375d : 00000000`00000001 00000000`00000000 000007fe`fe2b0000 00000000`775cf993 : rpcrt4!LRPC_SCALL::DispatchRequest+0x149

00000000`065af7a0 000007fe`fe2f09ff : 00000000`00000002 00000000`00000003 00000000`00000000 00000000`0418a2c0 : rpcrt4!LRPC_SCALL::HandleRequest+0x20d

00000000`065af8d0 000007fe`fe2f05b5 : 00000000`00000000 00000000`00000000 00000000`0049dc60 00000000`00000000 : rpcrt4!LRPC_ADDRESS::ProcessIO+0x3bf

00000000`065afa10 00000000`7760e2af : 00000000`0049dcd0 00000000`775c1ee4 00000000`05836930 00000000`05836930 : rpcrt4!LrpcIoComplete+0xa5

00000000`065afaa0 00000000`775cf61f : 00000000`0049dcd0 00000000`00000000 00000000`00000000 00000000`0000ffff : ntdll!TppAlpcpExecuteCallback+0x2cd

00000000`065afb10 00000000`7703652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x3f8

00000000`065afe10 00000000`775dba01 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd

00000000`065afe40 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

FOLLOWUP_IP:

upprov!InitializePrintProvidor+b87a4

000007fe`de8a9c84 4883c428 add rsp,28h

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: upprov!InitializePrintProvidor+b87a4

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: upprov

IMAGE_NAME: upprov.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 541d013a

STACK_COMMAND: ~28s ; kb

BUCKET_ID: X64_HANG_upprov!InitializePrintProvidor+b87a4

FAILURE_BUCKET_ID: APPLICATION_HANG_DEADLOCK_cfffffff_upprov.dll!InitializePrintProvidor

WATSON_STAGEONE_URL: http://watson.microsoft.com/00000000.htm?Retriage=1

Followup: MachineOwner

---------

Locking threads :

0:000> !locks

CritSec +4d554f8 at 0000000004d554f8

WaiterWoken No

LockCount 167

RecursionCount 1

OwningThread 878

EntryCount 0

ContentionCount f7

*** Locked

CritSec +4dc5a98 at 0000000004dc5a98

WaiterWoken No

LockCount 0

RecursionCount 1

OwningThread 878

EntryCount 0

ContentionCount 0

*** Locked

CritSec +4dc93d8 at 0000000004dc93d8

WaiterWoken No

LockCount 0

RecursionCount 1

OwningThread 878

EntryCount 0

ContentionCount 0

*** Locked

CritSec +4db3fe8 at 0000000004db3fe8

WaiterWoken No

LockCount 1

RecursionCount 1

OwningThread 878

EntryCount 0

ContentionCount 1

*** Locked

CritSec +4dc54e8 at 0000000004dc54e8

WaiterWoken No

LockCount 1

RecursionCount 1

OwningThread 841c

EntryCount 0

ContentionCount 2

*** Locked

Relevant thread-stacks :

0:000> ~~[878] kv

Child-SP RetAddr : Args to Child : Call Site

00000000`03b6f988 00000000`775fd7a8 : 00000000`00000001 00000000`04d574b0 00000000`04dc5890 000007fe`de8d9edd : ntdll!ZwWaitForSingleObject+0xa

00000000`03b6f990 00000000`775fd69b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`04dc54e8 : ntdll!RtlpWaitOnCriticalSection+0xe8

00000000`03b6fa40 000007fe`de8a9c84 : 00000000`00000000 00000000`03b6fb90 00000000`03b6fb40 00000000`04df3ca0 : ntdll!RtlEnterCriticalSection+0xd1

00000000`03b6fa70 000007fe`de7fd618 : 00000000`00000000 000007fe`de8e5943 00000000`03b6fb30 00000000`04dab0b0 : upprov!InitializePrintProvidor+0xb87a4

00000000`03b6faa0 000007fe`de8d975c : 00000000`03b6faf0 00000000`04dc54e0 00000000`04dc54d0 00000000`04dab0b0 : upprov!InitializePrintProvidor+0xc138

00000000`03b6fad0 000007fe`de8e4737 : 00000000`04dc54d0 00000000`04dab101 00000000`00000000 000007fe`de8a9cd4 : upprov!InitializePrintProvidor+0xe827c

00000000`03b6fb10 000007fe`de8de98e : 00000000`04db3fa0 00000000`04dab101 00000000`00000000 000007fe`de7fd646 : upprov!InitializePrintProvidor+0xf3257

00000000`03b6fbd0 000007fe`de8d576d : 00000000`04dc9390 00000000`04dbe701 00000000`04dce8c8 000007fe`de8cd49d : upprov!InitializePrintProvidor+0xed4ae

00000000`03b6fca0 000007fe`de8d5834 : 00000000`04dc5a70 00000000`04dbb601 00000000`00000001 00000000`04d55540 : upprov!InitializePrintProvidor+0xe428d

00000000`03b6fd50 000007fe`de8c8032 : 00000000`04dc5a70 00000000`04dbe601 00000000`00000000 00000000`000006cc : upprov!InitializePrintProvidor+0xe4354

00000000`03b6fdc0 000007fe`de8c7dc5 : 00000000`04d551e0 00000000`00000000 00000000`00000000 00000000`00000000 : upprov!InitializePrintProvidor+0xd6b52

00000000`03b6fed0 00000000`7703652d : 00000000`04d551e0 00000000`00000000 00000000`00000000 00000000`00000000 : upprov!InitializePrintProvidor+0xd68e5

00000000`03b6ff10 00000000`775dba01 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd

00000000`03b6ff40 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

0:000> ~~[841c] kv