Folks, Prepare Your Environment for the Holidays!

Happy Thanksgiving, folks!

In today’s HOW-TOs edition, I would like to encourage you to prepare your environment for the holiday weekend if you haven’t done so yet. A nice way to keep an eye on your infrastructure is to set up an alerting system for you to be alarmed via email if an unexpected event occurs. One way to configure proactive monitoring is to attach a task to an Event ID in Windows Event Viewer and tell Windows to send you an email every time that Event ID occurs. Here is a list of IDs with context from sources “StreamService,” “StreamProcess” and “Service Control Manager” that may serve you well when it comes to your PVS servers.

Application Log:

1. Event ID 1 – PVS Stream Service STARTED. <Information>

2. Event ID 101 – Unable to contact Citrix license server. <Error>

3. Event ID 107 – Licensing grace period expired. <Error>

4. Event ID 11 – DBAccess error. <Error>

5. Event ID 11 (again) – StreamProcess IP <IP:Port> is disconnected or non-functional. <Error>

System Log:

1. Event ID 7036 – The Citrix PVS Stream Service service entered in a stopping state. <Information>

2. Event ID 7034 – The Citrix PVS Stream Service service terminated unexpectedly. <Error>

For PVS targets, I recommend setting up an alert for any errors in the Event Viewer from source “Bnistack” as those normally point to retries and/or network disconnections. If you want to take it a step further, visit John Howard’s blog from Microsoft Technet for additional scripting techniques to get the most information from Windows Event Viewer via email alerts.

And last but not least, feel free to leverage WER (Windows Error Reporting) to generate crash dumps across the board or on a per-application basis whenever a relevant process decides to crash, find instructions HERE.

OK, everyone. The turkey is waiting! 🙂 Enjoy a safe weekend!

A Few Notes on the Load Balancing Algorithm

Isn’t it nice that we can load balance our vDisks across different servers in PVS and spread the load of connections? But how does load balancing occur and when?

The answer is during target device boot. After the target has acquired an IP address and has downloaded the bootstrap file from TFTP (or Two-Stage Boot in the case of BDM) it makes its first contact with the PVS farm by initiating a connection with the login server listed in the boostrap. After the login server determines the device is present in the database and part of a Device Collection within that farm, it uses a load balancing algorithm to calculate the number of active connections on each PVS server and hands the device over to the least busy one. Not too bad!

There is a subsetting of the load balancing properties on each vDisk that you can tweak from the PVS Console called Subnet Affinity. What Subnet Affinity does is prioritize the servers during load balancing based on the subnet they reside on. You have three configurations for Subnet Affinity at your disposal:

1. None

This one doesn’t take into account subnetting and hands the connection over to the least busy server regardless of network location.

2. Best Effort

Here the PVS server tries extra hard to keep the connection on a server in the same subnet. If not possible, it reaches out to the rest of the hosts in other subnets.

3. Fixed

In “Fixed” the Provisioning Server “forgets” other networks and all connections are handed out to hosts on the same subnet. If no one is available, load balancing doesn’t take place at all.

In Provisioning Services Console you also have the option to rebalance your devices manually or automatically using a pre-set % threshold.

Tip. NEVER use both Subnet Affinity and Auto Rebalance at the same time. Find out why HERE.

 

VHD

VHD (short for Virtual Hard Disk) is a file type spec that allows a hard drive system to be contained into an individual file. This file can be used as a boot partition on physical hosts and virtual machines. The VHD format is NOT proprietary to Citrix but is rather a Microsoft-maintained technology. We use it in PVS as a virtual disk image of an Operating System that a PVS target device can boot to from the network. For more information on VHD, visit this MSDN web resource.

VHD

Citrix Diagnostic Facility (CDF) & PVS 7

Many of us, logging geeks, are used to debugging problems in our infrastructure by looking at historical logs (which most every application nowadays has) to pinpoint these issues. Why? Because generally logs never lie. And people do.

If you deploy or manage a Citrix environment, you are also well aware by now how important logs are for troubleshooting regardless if you do it yourself or you call Citrix Support. In PVS 5.x and 6.x, if logging is enabled, you would normally see the following logs under C:\ProgramData\Citrix\Provisioning Services\Logs:

LogsPicture

Each log corresponded to a process. For instance, if you are troubleshooting an issue with the Console GUI you would be looking at Console.log, the stream process would log to StreamProcess.log, etc. In PVS 7.x, however, server-side historical logging is no longer enabled through PVS nor it’s located under the same directory. Our modules are now integrated in Citrix Diagnostic Facility (or CDF) much like XenApp, XenDesktop and Citrix Receiver/ICA Client. There are two options that you can use to generate and collect a single CDF trace from PVS:

1. CDF Control (download it from CTX111961)

This tool is nice and easy. All you need to do is download it on your PVS server, extract it, run it as an Administrator, and start a trace (preferably select all modules). This works great if your issue is reproducible at will (for example, a MAPI error when launching the PVS Console). The trace is generated as a .ETL file in the same folder you installed CDF Control and can be viewed from the same tool or parsed to a .CSV file to open in a more convenient program such as Excel.

2. CDF Monitor (download it from CTX129537)

This is a very, very powerful utility. A bit more complex to install but adds a real value to your environment. It runs as a Windows service and generates a circular CDF trace under C:\Windows\CDF Monitor. There is a config file that comes with it and for PVS there is an extra step to go to CTX138698 and download the PVS-specific config and place it in the CDF Monitor installation folder to replace the original one. Luckily, full instructions on how to deploy CDF Monitor are provided in the same article. I highly recommend this tool for capturing intermittent issues and root cause analysis of production outages.

Note: CDF tracing is only integrated with PVS 7.x. There are no CDF modules for previous versions. Target-side logging can still be enabled from the PVS Console and is logged into CDF. ConfigWizard and Console logs are still available under the old folder.

Enjoy the deep dive!

 

Who Is the PVS Guy?

ProfilePic

My name is Konstantin Cvetanov and I am a Technical Consultant in Atlanta, Georgia. I specialize in design and implementation of virtualization solutions for companies of all sizes and industries. Before my current role, I worked at Citrix as an Escalation Engineer fixing issues of high complexity, mitigating production outages, and finding bugs in the code. During my time at Citrix I specialized in the Provisioning Services product line and that’s when I made this website to share my experience with other IT enthusiasts out there. I’ve recently added new sections to talk about other products like XenDesktop and NetScaler, but make no mistake – PVS will always be my baby!

DISCLAIMER: The knowledge produced on this website is only for the purposes of helping the community through sharing personal experience and point of view and in no way does it represent any support statements from the company I work for nor it bears any responsibility from Citrix.

How to Apply a Cumulative Hotfix on Your PVS Servers?

Hello Folks,

In today’s HOW-TO edition we’ll cover a fairly simple but very important method: applying a cumulative hotfix that requires a full reinstall of PVS on your two HA configured Provisioning servers. Here are the steps:

1. Download a hotfix such as CPVS61019 from https://support.citrix.com.

2. Stop the Stream service on PVS01. Your targets should fail over to PVS02. Reinstall PVS Console and Server software from the hotfix package on PVS01.

3. Rerun the PVS Configuration Wizard. When presented with farm options, select “Farm is already configured.”

4. Breeze through the Config Wizard. Fortunately, the tool selects by default your previous configuration settings, so no need to change anything unless you really need to. If the process completes successfully, Soap and Stream services will be restarted automatically.

5. Stop the Stream service on PVS02. Your targets should fail over to PVS01. Repeat the same procedure 1-3 on PVS02. You can then rebalance your targets manually if needed.

6. You are done.

Note: I generally recommend scheduling a maintenance window for updating your servers. Even though the procedure can be finished in 10 minutes, you don’t want to find yourself in a situation where HA failover doesn’t work as expected and you lose half of your connections when stopping the services.

Easy wasn’t it? 🙂

 

About That New Cache Type…

As many of you have noticed, PVS 7.1 has a brand new cache type called “Cache in Device RAM with Overflow on Hard Disk.” This new feature of PVS is designed to provide better performance by combining the light speed of RAM with the efficiency of hard disk storage and at the same time avoiding previous hurdles such as unexpected BSOD when using RAM cache due to the memory getting filled up. The new differencing format of the file (VHDX) also resolves the issue when caching to device HD where applications accessing printer drivers would randomly crash.

As some of you have noticed, however, target device performance has not increased dramatically in terms of speed. In fact, some folks out there with IOMeters have reported that IOPs have not improved at all with the new cache type. This is currently a known issue due to a problem turning on the RAM portion of the cache and I know for a fact that Citrix is working on fixing it in the next hotfix release for PVS 7.1. So stay excited!

Edit:

The RAM portion of the this cache type is fixed in CTX140338 which is a target device hotfix.

Which AD Groups Have Access to Your Farm?

If you ever get locked out of your PVS Console because someone in your organization changed Active Directory membership groups around, you will need to find out what security groups have permission to access the farm. It’d be super-easy if you could just open your Console…Security tab under Farm Properties. But what if you lost access?

…Fortunately there is a way because it’s all in the database. All you need to do is login to your SQL server, launch SQL Management Studio, expand the PVS database, right-click on the dbo.AuthGroup table, select top 1000 rows, and you will see a list of all the AD user groups that have permissions to access the farm. Then you will most likely realize that your Console user is NOT a member of any of those groups!

AuthGroup

How to Reverse Image a vDisk?

Reverse imaging can be a tedious procedure for some of us but is necessary to keep our vDisks up-to-date with hypervisor tools and cumulative hotfixes for PVS. Here are my 14 steps:

01. Boot your target device to the vDisk you want to reverse image with additional disk attached (same size or larger than the vDisk).
02. Make sure the new disk is visible in Windows Disk Management and mounted as a drive (i.e. D:\).
03. Use BNImage or XenConvert to copy the vDisk to the added drive (both tools are located under C:\Program Files\Citrix\Provisioning Services).
04. Make sure the new volue is set to Active in Disk Management.
05. Set the device to boot from Hard Disk in the PVS console.
06. Boot the device to Hard Drive by manually changing the boot order in BIOS.
07. Remove any antivirus software (Reboot).
08. Remove PVS target device software (Reboot).
09. Update hypervisor tools if necessary (Reboot).
10. Install latest PVS target device software (Reboot).
11. Run the PVS Imaging Wizard (System will reboot automatically and will continue to finish the conversion process). Before reboot change the boot order in BIOS to Network boot.
12. Switch the device to boot from vDisk in the PVS console (in Private or Maintenance Mode).
13. Install Antivirus software.
14. You are done.

What are yours? 🙂

Hotfix 19 (CPVS61019)

CPVS61019 (widely known as Hotfix 19) is the most current patch for PVS 6.1. It contains both server and target binaries and it’s essentially a full reinstall of PVS. Yes, it does require reverse imaging to upgrade your targets but in turn you get a whole lotta fixes from Stream Process to HA failover and Console performance improvements. Highly recommended! Get it here.