Happy Monday, Everyone!
Since we are very early in the week, let’s talk about some failures very early in the boot process of a Provisioning Services target device. First of all, a quick refresher on the Preboot Execution Environment (PXE): it’s a protocol that enables a client machine to boot from it’s network interface and connect to a server resource on the network to retrieve a bootfile program and load an Operating System. When a workstation PXE boots, the PXE client sends a DISCOVER packet to the entire broadcast domain to search for a DHCP server to get an IP address. If it doesn’t find one, it tries a few times and then it times out. I want to underline that this has NOTHING to do with PVS as this target device is nothing more than a PXE-enabled machine that early in the boot process. Failure to obtain an IP due to DISCOVER packet never reaching a DHCP server is in most cases observed in a subnetted network where PVS targets are on one physical segment and DHCP resources are on a another one. Because of a limitation in PXE (so old yet so useful), the broadcast packet hits a stonewall at your router and it never reaches your other segments. How do you get around that? Fortunately, there is a feature called DHCP Relay (also known as IP Helper for specific vendors) that you can enable with a simple command on your router in order to make it “listen” to PXE packets and “relay” them to the next subnet, and the next subnet, and the next subnet until they reach a DHCP destination server.
For specific information on enabling DHCP Relay on Cisco routers, read THIS documentation from Cisco. There is also a nice Citrix article on the topic with screenshots.
Thanks for reading!
Happy Thanksgiving, folks!
In today’s HOW-TOs edition, I would like to encourage you to prepare your environment for the holiday weekend if you haven’t done so yet. A nice way to keep an eye on your infrastructure is to set up an alerting system for you to be alarmed via email if an unexpected event occurs. One way to configure proactive monitoring is to attach a task to an Event ID in Windows Event Viewer and tell Windows to send you an email every time that Event ID occurs. Here is a list of IDs with context from sources “StreamService,” “StreamProcess” and “Service Control Manager” that may serve you well when it comes to your PVS servers.
1. Event ID 1 – PVS Stream Service STARTED. <Information>
2. Event ID 101 – Unable to contact Citrix license server. <Error>
3. Event ID 107 – Licensing grace period expired. <Error>
4. Event ID 11 – DBAccess error. <Error>
5. Event ID 11 (again) – StreamProcess IP <IP:Port> is disconnected or non-functional. <Error>
1. Event ID 7036 – The Citrix PVS Stream Service service entered in a stopping state. <Information>
2. Event ID 7034 – The Citrix PVS Stream Service service terminated unexpectedly. <Error>
For PVS targets, I recommend setting up an alert for any errors in the Event Viewer from source “Bnistack” as those normally point to retries and/or network disconnections. If you want to take it a step further, visit John Howard’s blog from Microsoft Technet for additional scripting techniques to get the most information from Windows Event Viewer via email alerts.
And last but not least, feel free to leverage WER (Windows Error Reporting) to generate crash dumps across the board or on a per-application basis whenever a relevant process decides to crash, find instructions HERE.
OK, everyone. The turkey is waiting! 🙂 Enjoy a safe weekend!
Isn’t it nice that we can load balance our vDisks across different servers in PVS and spread the load of connections? But how does load balancing occur and when?
The answer is during target device boot. After the target has acquired an IP address and has downloaded the bootstrap file from TFTP (or Two-Stage Boot in the case of BDM) it makes its first contact with the PVS farm by initiating a connection with the login server listed in the boostrap. After the login server determines the device is present in the database and part of a Device Collection within that farm, it uses a load balancing algorithm to calculate the number of active connections on each PVS server and hands the device over to the least busy one. Not too bad!
There is a subsetting of the load balancing properties on each vDisk that you can tweak from the PVS Console called Subnet Affinity. What Subnet Affinity does is prioritize the servers during load balancing based on the subnet they reside on. You have three configurations for Subnet Affinity at your disposal:
This one doesn’t take into account subnetting and hands the connection over to the least busy server regardless of network location.
2. Best Effort
Here the PVS server tries extra hard to keep the connection on a server in the same subnet. If not possible, it reaches out to the rest of the hosts in other subnets.
In “Fixed” the Provisioning Server “forgets” other networks and all connections are handed out to hosts on the same subnet. If no one is available, load balancing doesn’t take place at all.
In Provisioning Services Console you also have the option to rebalance your devices manually or automatically using a pre-set % threshold.
Tip. NEVER use both Subnet Affinity and Auto Rebalance at the same time. Find out why HERE.
VHD (short for Virtual Hard Disk) is a file type spec that allows a hard drive system to be contained into an individual file. This file can be used as a boot partition on physical hosts and virtual machines. The VHD format is NOT proprietary to Citrix but is rather a Microsoft-maintained technology. We use it in PVS as a virtual disk image of an Operating System that a PVS target device can boot to from the network. For more information on VHD, visit this MSDN web resource.
Many of us, logging geeks, are used to debugging problems in our infrastructure by looking at historical logs (which most every application nowadays has) to pinpoint these issues. Why? Because generally logs never lie. And people do.
If you deploy or manage a Citrix environment, you are also well aware by now how important logs are for troubleshooting regardless if you do it yourself or you call Citrix Support. In PVS 5.x and 6.x, if logging is enabled, you would normally see the following logs under C:\ProgramData\Citrix\Provisioning Services\Logs:
Each log corresponded to a process. For instance, if you are troubleshooting an issue with the Console GUI you would be looking at Console.log, the stream process would log to StreamProcess.log, etc. In PVS 7.x, however, server-side historical logging is no longer enabled through PVS nor it’s located under the same directory. Our modules are now integrated in Citrix Diagnostic Facility (or CDF) much like XenApp, XenDesktop and Citrix Receiver/ICA Client. There are two options that you can use to generate and collect a single CDF trace from PVS:
1. CDF Control (download it from CTX111961)
This tool is nice and easy. All you need to do is download it on your PVS server, extract it, run it as an Administrator, and start a trace (preferably select all modules). This works great if your issue is reproducible at will (for example, a MAPI error when launching the PVS Console). The trace is generated as a .ETL file in the same folder you installed CDF Control and can be viewed from the same tool or parsed to a .CSV file to open in a more convenient program such as Excel.
2. CDF Monitor (download it from CTX129537)
This is a very, very powerful utility. A bit more complex to install but adds a real value to your environment. It runs as a Windows service and generates a circular CDF trace under C:\Windows\CDF Monitor. There is a config file that comes with it and for PVS there is an extra step to go to CTX138698 and download the PVS-specific config and place it in the CDF Monitor installation folder to replace the original one. Luckily, full instructions on how to deploy CDF Monitor are provided in the same article. I highly recommend this tool for capturing intermittent issues and root cause analysis of production outages.
Note: CDF tracing is only integrated with PVS 7.x. There are no CDF modules for previous versions. Target-side logging can still be enabled from the PVS Console and is logged into CDF. ConfigWizard and Console logs are still available under the old folder.
Enjoy the deep dive!
My name is Konstantin Cvetanov and I am a Technical Consultant in Atlanta, Georgia. I specialize in design and implementation of virtualization solutions for companies of all sizes and industries. Before my current role, I worked at Citrix as an Escalation Engineer fixing issues of high complexity, mitigating production outages, and finding bugs in the code. During my time at Citrix I specialized in the Provisioning Services product line and that’s when I made this website to share my experience with other IT enthusiasts out there. I’ve recently added new sections to talk about other products like XenDesktop and NetScaler, but make no mistake – PVS will always be my baby!
DISCLAIMER: The knowledge produced on this website is only for the purposes of helping the community through sharing personal experience and point of view and in no way does it represent any support statements from the company I work for nor it bears any responsibility from Citrix.
In today’s HOW-TO edition we’ll cover a fairly simple but very important method: applying a cumulative hotfix that requires a full reinstall of PVS on your two HA configured Provisioning servers. Here are the steps:
1. Download a hotfix such as CPVS61019 from https://support.citrix.com.
2. Stop the Stream service on PVS01. Your targets should fail over to PVS02. Reinstall PVS Console and Server software from the hotfix package on PVS01.
3. Rerun the PVS Configuration Wizard. When presented with farm options, select “Farm is already configured.”
4. Breeze through the Config Wizard. Fortunately, the tool selects by default your previous configuration settings, so no need to change anything unless you really need to. If the process completes successfully, Soap and Stream services will be restarted automatically.
5. Stop the Stream service on PVS02. Your targets should fail over to PVS01. Repeat the same procedure 1-3 on PVS02. You can then rebalance your targets manually if needed.
6. You are done.
Note: I generally recommend scheduling a maintenance window for updating your servers. Even though the procedure can be finished in 10 minutes, you don’t want to find yourself in a situation where HA failover doesn’t work as expected and you lose half of your connections when stopping the services.
Easy wasn’t it? 🙂