Adding vCloud Air as a vCloud Automation Center Application Services Cloud Provider

Long title, huh? Hooray very long product names.

If you’re planning on adding a vCloud Air instance as a cloud provider for Application Services, there are a few steps that aren’t covered in the Application Services documentation.

  1. In order to create the cloud provider, you’ll need to retrieve the URL & organization name from the vCloud Air user interface. Login to vCloud Air and click on the organizational virtual datacenter from the Dashboard tab.
  2. On the right side of the virtual datacenter details page, click the Cloud Provider Address under Related Links on the rights side of the page. Grab the URL and organization name.
  3. Login to Application Services as an application cloud administrator and go to add a new cloud provider.
  4. Choose the vCloud provider type. Enter the FQDN, minus the https:// and port 443 that was shown in the vCloud Air UI. Enter the organization name. Enter the correct user name and password from an active user within vCloud Air. Choose the appropriate vCAC Business Group.
  5. The documentation states that the @ symbol in the user’s email address should be replaced with %40. However, Application Services should do this automatically.
  6. Validate and save.

Environment:

  • vCloud Automation Center Application Services 6.1 Build 2064245

vCloud Automation Center HTTP 503 error

The other day I logged into my vCloud Automation Center web client and discovered that I was getting HTTP 503 errors in a number of different places and Java errors thrown in my Home dashboard. After a bit of digging around I determined that the password expired on the Active Directory user account that’s acting as a service account. The vCloud Automation Center IaaS services run as a domain user, and if the password expires the services won’t start and everything breaks.

The fix is, of course, to change the password on the user account in Active Directory, change the password on the services, and start everything back up. It’s possible (I think) to use an Active Directory managed service account for those services, but that doesn’t help all the places that use that service account, since they’re not “smart” enough to do so. Note: My hesitation about whether or not those Windows services can use a managed service account isn’t because of Windows, but because I don’t know if the IaaS installer would object to dollar signs in the user name. I haven’t had a chance to try it as I’d need to spin up another VM for IaaS install and… /sigh.

So where does that leave us? Back to the ol’ user account pretending to be a service account. Yay! Everyone loves manually managing password rotations, change management windows to bounce services, or applying group policies to an OU in order to make a domain password policy exception, right? ūüėČ

You’ll need to change the password on the following services: VMware DEM-Orchestrator, VMware DEM-Worker, VMware vCloud Automation Center Agent, and VMware vCloud Automation Center Service. Be sure to¬†change the password on the following IIS Application Pools: RepositoryAppPool, vCACAppPool, and WapiAppPool.

Moral of the story:¬†if IT was easy, it wouldn’t pay so well.

HTTP503-1
HTTP503-2
HTTP503-3
Environment:

  • vCloud Application Director 6.0.1.0 Build 1571033
  • vCloud Automation Center 6.0.1 Build: 1571033, using vCenter’s SSO
  • vCenter Server Appliance 5.5.0.10100 Build 1750781

Setting up the CentOS 6.3 template for jPetStore in vCloud Application Director 6.0.1

If you’ve tried to use the CentOS 6.3 template that VMware provides as part of the out-of-the-box application templates for vCloud Application Director 6.0.1, then you’re probably aware of the difficulty involved due to sub-optimal documentation and the template’s initial configuration. I’ve worked through the documentation and after a bit of work and head scratching, I was able to successfully deploy the jPetStore sample application. Here are the steps from start to finish.

Prepare the CentOS 6.3 template
  1. After downloading the CentOS 6.3 .ovf, .mf, and .vmdk file, deploy the ovf to¬†your destination vSphere host using the vCenter web or thick client. Make sure you’re deploying it to the cluster(s) that are assigned to the appropriate business group within vCloud Automation Center. ¬†Take a snapshot here, in case you need to revert to the original template.
  2. Edit the newly deployed VM and verify that DHCP is visible under vApp Options > Deployment.
  3. Power on the VM, and open the VM console. Logon using username root and password vmware. You’ll need to change the contents of two files as they contain left over network information from eng.vmware.com.
  4. Make a backup of resolv.conf and then empty the file. This will force the VM to use the DNS settings provided by your network DHCP.
    cp /etc/resolv.conf resolv.conf.bak
    cat /dev/null > /etc/resolv.conf
    
  5. Second, backup /etc/sysconfig/network and make changes so eth0 will use DHCP when it comes up.
    cp /etc/sysconfig/network /etc/sysconfig/network.bak
    cat > /etc/sysconfig/network
    DEVICE=eth0
    NETWORKING=yes
    BOOTPROTO=dhcp
    ONBOOT=yes
    CTRL+C
    
  6. If you have an existing vCloud Application Director agent bootstrap service, uninstall the service.
    1. Check to see if the agent is running and stop it if it is.
      ps -ef | grep agent_bootstrap
      kill -9 <process identified>
    2. Run the Shell file to remove the agent bootstrap service.
      /opt/vmware-appdirector/agent-bootstrap/agent_reset.sh
    3. Uninstall the agent bootstrap service
      rpm -e vmware-appdirector-agent-service-6.0.0.0-0.i386
  7. Download and install the vCloud Automation Center guest agent. (The docs are incorrect here.)
    1. Open a browser window and go to¬†https://<IP or FQDN of vCAC appliance>:5480/installer. Download the Linux guest agent packages & scp them to the CentOS VM (If you’re running Windows, you can use WinSCP) or from the VM:
      wget --no-check-certificate https://<IP or FQDN of vCAC appliance>:5480/installer/LinuxGuestAgentPkgs.zip

      Note: –no-check-certificate is needed when hitting an https link and you’re using self-signed certificates.

    2. If you used wget or SCP’d the entire zip file to root’s home, use unzip.
      unzip -d LinuxGuestAgentPkgs LinuxGuestAgentPkgs.zip

      Note: the -d flag will create a folder named LinuxGuestAgentPkgs and unzip there.

      cd LinuxGuestAgentPkgs/rhel6-x86
      rpm -i gugent-6.0.1-71.i386.rpm
  8. Download and install the Application Director agent bootstrap service.
    wget --no-check-certificate http://p10-l1-appd01.dooleylab.dns/agent/vmware-appdirector-agent-service-vcac_6.0.0.0-0_i386.rpm
    rpm -i vmware-appdirector-agent-service-vcac_6.0.0.0-0_i386.rpm
  9. Register the Application Director agent bootstrap service to the vCloud Automation Center server.
    /opt/vmware-appdirector/agent-bootstrap/vcac-register.sh -r vCAC_Port -s IaaS_Server_FQDN

    Note: You may see errors in the output saying that it’s unable to get the local issuer certificate, however it should still work, as I received those errors but still successfully deployed the template.

  10. Verify that the vrm-agent and vmware_appdirector_agent services are set to on for runlevel 3.
    chkconfig --list | grep agent
  11. Verify that the dmidecode-2.11-2.el6.i686 rpm is installed. (It should be already.)
    rpm -qa | grep dmi
  12. Register and run the vCloud Automation Center agent.
    /usr/share/gugent/installgugent.sh vCAC_FQDN_or_IP:443 ssl
    /usr/share/gugent/rungugent.sh

    You should receive a number of application debug messages. Providing no error messages are returned, you’re good to go.

  13. Shutdown the VM and take another snapshot.
Create the vCloud Automation Center blueprint
  1. Chances are that the data collection processes haven’t run on the vCAC server yet, so login and go to Infrastructure > Compute Resources > Compute Resources. Hover over the compute resource that contains the CentOS VM and click on Data Collection. Request the Inventory, State, and Performance data collection processes. Click Refresh to see their progress.
  2. Create the blueprint using the Clone workflow, making sure to select the latest snapshot that was taken on the CentOS template VM.
Update the CentOS 6.3 logical template
  1. Login to vCloud Application Director using a vCloud Automation Center business group user that has a minimum of Application Architect, Application Catalog Administrator, Application Cloud Administrator, Application Publisher and Deployer roles assigned in vCloud Automation Center Administration > Groups.
    Note: If¬†you’re using an Active Directory identity source and didn’t enter a domain alias, you’ll need to use the fully qualified domain name.
  2. Create a cloud provider if one hasn’t been created already, and add the CentOS 6.3 template (blueprint).
    Note: If you get a credentials error, see this¬†post about case sensitivity depending on the type of tenant identity source you’re using.
  3. Go to Logical Templates > CentOS 6.3 32bit > Logical Template Versions > Edit > New Cloud Template Mapping. Choose the registered cloud provider and CentOS 6.3 cloud template, then save.
  4. Go to Deployment Environments > Create a Deployment Environment > give it a name, choose the registered cloud provider.
Deploy the jPetStore application
  1. Go to Applications > jPetStore > Application Versions > Create Deployment Profile, give it a name, then deploy.
    1. Step 1: Deployment Environment – click Map Details and make sure that the Node Name is jPetStore, Logical Template is CentOS63 32bit, and Cloud is the CentOS 6.3 x32 template.
    2. Step 2: Application Properties – You can leave the vCPU and Memory property values blank and they’ll use the values defined in the jPetStore application blueprint, providing you have a minimum of 2048MB defined in the vCAC blueprint or a maximum equal to or more than 2048MB. If you have 1024MB defined as the minimum and maximum and don’t specify 1024MB as the new memory property value, deployment will fail. You can choose to enter a hostname, otherwise Application Director will give is a random name starting with jPetSt. The blueprint’s machine prefixes have no effect here.
    3. Step 3: Execution Plan – there’s nothing to do on this page¬†for the out of the box version of jPetStore.
    4. Step 4: Review – You can choose to Publish first, which will add this deployment in vCAC’s Catalog Items in the appropriate business group scope. If you don’t want to publish it to vCAC, click Deploy. Note: You’ll still need add it to the appropriate service for it to be made available to an entitled vCAC user.
  2. After the deployment completes successfully, grab the VM’s IP address from the deployment details. Navigate to http://jPetStore_VM_IP:8080/jpetstore-1.0.0 and enjoy jPetStore in all of its glorious glory. ūüôā

Environment:

  • vCloud Application Director 6.0.1.0 Build 1571033
  • vCloud Automation Center 6.0.1 Build: 1571033, using vCenter’s SSO
  • vCenter Server Appliance 5.5.0.10100 Build 1750781
  • CentOS 6.3 32 bit template dated 2013-12-10

Active Directory tenant identity sources & vCloud Application Director 6.0.1

Recently, I encountered an issue while trying to add a cloud provider in vCloud Application Director 6.0.1. It turns out that, depending on the vCloud Automation Center identity source, you may need to capitalize the domain name in the user’s UPN. (See this post about attempting to add a cloud provider with a down-level username vs. a UPN.)

If the vCAC tenant’s identity source is Active Directory (not Native Active Directory), you can login using the username format of username@fqdn-domain. If the identity source is Native Active Directory, you must use the username format of username@FQDN-DOMAIN. So, damian@lab.dns vs. damian@LAB.DNS. As you can see below, the SAML response returned from vCenter SSO has the domain name capitalized when using Native Active Directory, and not capitalized when using Active Directory.

This only happens when using the default vsphere.local tenant in vCloud Automation Center, as additional tenants can only use Open LDAP or Active Directory.

Adding a cloud provider using Active Directory identity source

Log from vCenter 5.5.0U1a’s SSO vmware-identity-sts.log, located at /var/log/vmware/sso:

TRACE com.vmware.identity.sts.auth.impl.UNTAuthenticator] Starting to authenticate principal: damian.karlson@lab.dns
DEBUG com.vmware.identity.sts.auth.impl.UNTAuthenticator] Authenticated principal: {Name: damian.karlson, Domain: lab.dns} at time: Thu Jun 26 16:50:57 UTC 2014
DEBUG com.vmware.identity.sts.impl.STSImpl] Authenticated principal: {Name: damian.karlson, Domain: lab.dns} at time: Thu Jun 26 16:50:57 UTC 2014
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] 5 attributes retrieved for {Name: damian.karlson, Domain: lab.dns}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=givenName, value=[Damian]] retrieved for {Name: damian.karlson, Domain: lab.dns}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://rsa.com/schemas/attr-names/2009/01/GroupIdentity, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=Groups, value=[lab.dns\Domain Admins, lab.dns\vCenter Admins, lab.dns\Domain Users, lab.dns\Denied RODC Password Replication Group, vsphere.local\Administrators, vsphere.local\Everyone]] retrieved for {Name: damian.karlson, Domain: lab.dns}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://schemas.xmlsoap.org/claims/UPN, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=userPrincipalName, value=[damian.karlson@lab.dns]] retrieved for {Name: damian.karlson, Domain: lab.dns}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=surname, value=[Karlson]] retrieved for {Name: damian.karlson, Domain: lab.dns}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://vmware.com/schemas/attr-names/2011/07/isSolution, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=Subject Type, value=[false]] retrieved for {Name: damian.karlson, Domain: lab.dns}

XML SAML response snipped farther down in the log:

<saml2:Subject>
	<saml2:NameID Format="http://schemas.xmlsoap.org/claims/UPN">damian.karlson@lab.dns</saml2:NameID>
	<saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:holder-of-key">
		<saml2:SubjectConfirmationData xsi:type="saml2:KeyInfoConfirmationDataType">
			<ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
				<ds:X509Data>
					<ds:X509Certificate>snipped</ds:X509Certificate>
				</ds:X509Data>
			</ds:KeyInfo>
		</saml2:SubjectConfirmationData>
	</saml2:SubjectConfirmation>
</saml2:Subject>

Log from vCloud Application Director’s catalina.out, located at /home/darwin/tcserver/darwin/logs/catalina.out:

INFO com.vmware.darwin.service.catalog.CloudProviderHelper - Logging in cloud url=https://iaas01.lab.dns with user=damian.karlson@lab.dns
INFO com.vmware.darwin.flow.engine.CloudServiceConnectionImpl - Generating SAML for user='damian.karlson@lab.dns' 
INFO com.vmware.darwin.cal.api.CALSessionFactory - Found cloud driver: type = vcac6_0 class = com.vmware.darwin.cal.driver.vcac6_0.VCACBackCloudDriver
INFO com.vmware.darwin.cal.driver.vcac6_0.VCACHelper - VCACCloudDriver instantiating...
INFO com.vmware.darwin.cal.driver.vcac6_0.VCACHelper - VCACCloudDriver instantiated successfully!

Adding a cloud provider using Native Active Directory identity source

Log from vCenter 5.5.0U1a’s SSO vmware-identity-sts.log:

TRACE com.vmware.identity.sts.auth.impl.UNTAuthenticator] Starting to authenticate principal: damian.karlson@lab.dns
DEBUG com.vmware.identity.sts.auth.impl.UNTAuthenticator] Authenticated principal: {Name: damian.karlson, Domain: LAB.DNS} at time: Thu Jun 26 18:53:22 UTC 2014
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] 5 attributes retrieved for {Name: damian.karlson, Domain: LAB.DNS}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=givenName, value=null] retrieved for {Name: damian.karlson, Domain: LAB.DNS}
com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=surname, value=null] retrieved for {Name: damian.karlson, Domain: LAB.DNS}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://vmware.com/schemas/attr-names/2011/07/isSolution, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=Subject Type, value=[false]] retrieved for {Name: damian.karlson, Domain: LAB.DNS}
com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://rsa.com/schemas/attr-names/2009/01/GroupIdentity, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=Groups, value=[lab.dns\Domain Users, lab.dns\vCenter Admins, lab.dns\Domain Admins, lab.dns\Denied RODC Password Replication Group, vsphere.local\Administrators, vsphere.local\Everyone]] retrieved for {Name: damian.karlson, Domain: LAB.DNS}
TRACE com.vmware.identity.saml.idm.IdmPrincipalAttributesExtractor] An attribute PrincipalAttribute [name=http://schemas.xmlsoap.org/claims/UPN, format=urn:oasis:names:tc:SAML:2.0:attrname-format:uri, friendly name=userPrincipalName, value=[damian.karlson@LAB.DNS]] retrieved for {Name: damian.karlson, Domain: LAB.DNS}

XML SAML response snipped farther down in the log:

<saml2:Subject>
	<saml2:NameID Format="http://schemas.xmlsoap.org/claims/UPN">damian.karlson@LAB.DNS</saml2:NameID>
	<saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:holder-of-key">
		<saml2:SubjectConfirmationData xsi:type="saml2:KeyInfoConfirmationDataType">
			<ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
				<ds:X509Data>
					<ds:X509Certificate>snipped</ds:X509Certificate>
				</ds:X509Data>
			</ds:KeyInfo>
		</saml2:SubjectConfirmationData>
	</saml2:SubjectConfirmation>
</saml2:Subject>

Log from vCloud Application Director’s catalina.out:

INFO com.vmware.darwin.service.catalog.CloudProviderHelper - Logging in cloud url=https://iaas01.lab.dns with user=damian.karlson@lab.dns
INFO com.vmware.darwin.flow.engine.CloudServiceConnectionImpl - Generating SAML for user='damian.karlson@lab.dns' 
ERROR com.vmware.darwin.flow.engine.CloudServiceConnectionImpl - Error occured while generating SamlToken for the user=damian.karlson@lab.dns
com.vmware.darwin.csp.exception.CspException: Use damian.karlson@LAB.DNS for Username field instead of damian.karlson@lab.dns

Environment:

  • vCloud Application Director 6.0.1.0 Build 1571033
  • vCloud Automation Center 6.0.1 Build: 1571033, using vCenter’s SSO
  • vCenter Server Appliance 5.5.0.10100 Build 1750781

Adding a cloud provider in vCloud Application Director 6.0.1

If you’ve tried to add a new cloud provider within vCloud Application Director 6.0.1, you may have run into an issue where the credentials can’t be validated. The (less than helpful) UI error message is: Could not connect to the Cloud Provider at https://iaas01.lab.dns: Unable to login to cloud provider. Please verify the user credentials as well as other parameters you entered.

The documentation says to use a down-level logon in the format of domain\username, which is incorrect. [Source: http://pubs.vmware.com/appdirector-6/topic/com.vmware.appdirector6.using.doc/GUID-0D61812D-98F7-4D0C-83C1-AF55678EFA53.html]

If you examine Application Director’s tcserver catalina.out log, located at /home/darwin/tcserver/darwin/logs/, you’ll see the following (surprisingly helpful!) error message.

Jun 24 2014 17:18:40.703 INFO  [http-bio-8443-exec-10] [damian.karlson@lab.dns] com.vmware.darwin.service.catalog.CloudProviderHelper - Logging in cloud url=https://iaas01.lab.dns with user=lab.dns\damian.karlson
Jun 24 2014 17:18:40.703 INFO  [http-bio-8443-exec-10] [damian.karlson@lab.dns] com.vmware.darwin.flow.engine.CloudServiceConnectionImpl - Generating SAML for user='lab.dns\damian.karlson' 
Jun 24 2014 17:18:42.134 ERROR [http-bio-8443-exec-10] [damian.karlson@lab.dns] com.vmware.darwin.flow.engine.CloudServiceConnectionImpl - Error occured while generating SamlToken for the user=lab.dns\damian.karlson
com.vmware.darwin.csp.exception.CspException: Use damian.karlson@lab.dns for Username field instead of lab.dns\damian.karlson

Environment:

  • vCloud Application Director 6.0.1.0 Build 1571033
  • vCloud Automation Center 6.0.1 Build: 1571033, using vCenter’s SSO
  • vCenter Server Appliance 5.5.0.10100 Build 1750781