Monday, 24 December 2012

Digital Signatures and Install Software gotcha

Problem: In automating SQL Server and SharePoint images, the actual installation is taking a long time on my managed environment whereas my developer laptop is fast.  All installations are done without Internet access (offline).

I have a dev environment build on my laptop that runs SSD and I run 3 VM using VMware workstation 9 (all use Windows 2008 R2 SP1).  I create an 1) AD with 1GB or RAM and 1 CPU 2) SQL 20012 with 10GB RAM and 4 CPU's 3) SP2010 CU Aug 2012 10GB 4 CPU's.  All the installation is automated using slip streamed images.

So for simplicity on the CI environment I will explain a simplified comparable setup. I have 3 machines with the same roles however the SQL 2012 and SP2010 install take considerably longer.  The CI environment is on ESX (Cisco blades & chassis, and Violin (SSD) storage.  The CPU/compute is connected to the storage via SAS/Fibre channel made no difference either).  I have summarised the results below:

                                                                 SQL2012 (duration)      SP2010 (duration)
Laptop(VMworkstation Workstation)           15 min                              16 min
CI (ESX)                                                        22 min                              92 min 

Finding: My hardcore/good ESX infrastructure is taking 9 minutes longer to install SQL Server 2012 on beter hardware and an amazing 70 minutes longer to install SP2010.

Update 21 Feb 2013: Don't use PowerShell 3 with AutoSPInstaller (including using the version switch i.e. -version 2), it doesn't work and even changing AutoSPInstallers internal web call fail.  It can be made to work with the version 2 switch but it isn't worth the effort.

Initial Hypothesis:
After many many hours between service providers managing the infrastructure, it was not hardware or ESX configuration/setup.  However if the network card on the VM is disabled, the performance change improves to:

                                                          SQL2012 (duration)            SP2010 (duration)
CI (ESX)                                                     13 min                       5 min and 5 seconds

Pretty hefty improvement.  Using netstat is looks like there are requests to the Internet.  After adding Wireshark to monitor all traffic.  I can see requests being sent to crl.mirosoft.com (certificate revocation lists) and ctldl.windowsupdate.com

Issue shown in Wireshark
Issue Shown in Fiddler
This is the 1st time I have seen this issue in a clients production environment.  If the WFEs/SP servers have internet access (less preferable) or the servers don't have access the install work in a timely fashion.  The symtoms of the issue are when the WFE's/SP Servers don't have internet access but think they do.  All the binaries are digitally signed and the install will try validate the signatures despite this being an offline install.

I confirmed the problem being how the networking is setup.  My issue shows up on the VM NIC adapter, Originally the IPv4 Connectivity has a status of "No Internet Access", once I ping google I get a reply and the status changes to "Internet".  I can ping google but not browse to it.


Resolution:  The problem is that executable code is digitally signed.  This is good, all code should be digitally signed so it can be authenticated.  However in this situation a lot of requests are being sent out from the VM as the install tries to verify all the SharePoint complied code.  The install on the local VM acts as if there is an Internet connection (which there is not).

It takes unique networking to get into this issue and SP/any digitally signed code will check the digital certs.

There are a few fixes such as:
1.> Allowing the servers to get out to the Internet, so open the firewall or set a proxy on the local VM.
2.> Add host entries to the cert fails immediately but will continue installing (This is not working for me).
3.> Make the following registry change:
set-ItemProperty -path "HKCU:\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944
set-ItemProperty -path "REGISTRY::\HKEY_USERS\.Default\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing" -name State -value 146944
get-ChildItem REGISTRY::HKEY_USERS | foreach-object {set-ItemProperty -ErrorAction silentlycontinue -path ($_.Name + "\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing") -name State -value 146944}


More Information:

Certificates for installing sofware is cause slow install:
http://joelblogs.co.uk/2011/09/20/certificate-revocation-list-check-and-sharepoint-2010-without-an-internet-connection/

http://ddkonline.blogspot.co.uk/2010/05/fix-sharepoint-very-slow-to-start-after.html

If you want to verify if a machine is having problems with a poarticlar process Process Explorer (Usefule if a machine has high memory, CPU or IO issues)

1 comments:

Digital Signature said...
This comment has been removed by a blog administrator.

Post a Comment