Last saturday we upgraded our vCloud Director to version 5.5.3 and vShield manager to 5.5.4.1.

The fact is after the upgrade vApp operations (poweron, poweroff) are much slower than before. The sluggishness is not happening always but when it does, vCloud logs are filled with the following messages:

2015-07-08 09:24:01,466 | WARN     | ValidationThread          | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,475 | WARN     | pool-validationservice-13-thread-7237 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,479 | WARN     | pool-validationservice-13-thread-7232 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,480 | WARN     | pool-validationservice-13-thread-7235 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,485 | WARN     | ValidationThread          | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,503 | WARN     | pool-validationservice-13-thread-7235 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,503 | WARN     | pool-validationservice-13-thread-7232 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,504 | WARN     | pool-validationservice-13-thread-7237 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,508 | WARN     | ValidationThread          | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,510 | WARN     | pool-validationservice-13-thread-7231 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:01,523 | WARN     | pool-validationservice-13-thread-7237 | ReactionServiceImpl            | Dropping reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b, 10 of 10 reactions already running. |

2015-07-08 09:24:31,259 | WARN     | pool-validationservice-13-thread-7232 | ReactionServiceImpl            | Scheduling reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b for execution. |

2015-07-08 09:24:31,260 | WARN     | pool-validationservice-13-thread-7236 | ReactionServiceImpl            | Scheduling reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b for execution. |

2015-07-08 09:24:31,278 | WARN     | pool-validationservice-13-thread-7236 | ReactionServiceImpl            | Scheduling reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b for execution. |

2015-07-08 09:24:31,312 | WARN     | pool-validationservice-13-thread-7232 | ReactionServiceImpl            | Scheduling reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b for execution. |

2015-07-08 09:24:31,313 | WARN     | pool-validationservice-13-thread-7236 | ReactionServiceImpl            | Scheduling reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b for execution. |

2015-07-08 09:24:31,316 | WARN     | pool-validationservice-13-thread-7235 | ReactionServiceImpl            | Scheduling reaction com.vmware.vcloud.backend.reactions.SyncCrosshostWithHostReaction@45b7601b for execution. |

Support has told us to lower vcenter statistics level (currently 3) and to delete the content of inventory tables from vCloud director database but the problem persists after doing that.

Any help?

Didnt find anything specific in the query result. today i again investigate the storage as i thought there must be something wrong with the storage. as i can see there are multiple extents in to that.

They are like 2 TB in starting(may be upgraded to vmfs5) and 10 TB later.

I also opened a SR in vmware to see if they can help .

Image may be NSFW.
Clik here to view.

we had a similar issue.  not exactly what you are describing, but close.  you may want to check via vcenter web client that your storage profile contains the tag referencing your volume.  It looked correct in the normal vSphere client but in the web client, it was not there.

i created the tags and profile twice from webclient only. even i cant see those profile from the vsphere client. but visible in vsphere client.

Some things are super funny with the vmware.

Thanks IamTHEvilONE& ccinatl

Today i fixed the issue from my self. Dont know what was wrong. but i removed old storage profile tags etc from vcloud director and vsphere itself. created a new one. and now its working superawesome.

I forgot to tell you the reason. as i suspect : I created the storage profile 1 year ago and till that date the datastore frequently updated with the size and the name both. i added that storage profile 2 days ago . may be thats the reason but not anymore.

Cheers.

Hello,

  We are doing a migration of vms from vSphere to vCloud director , by using import from vSphere in vCD . We are selecting Move VM , therefore keeping the UUID / MAC addreses etc. What would be a backout solution in case we have to revert back to vSphere ? So far, I only found a solution to modify the VMX , readd the VM in the Inventory and deleting the vApp from vCloud director , which I'm not very happy . Is there an easier solution ?

  I was thinking of selecting Copy instead of Move , when importing from vSphere . I've done a test and noticed that it actually keeps the bios.uuid and mac addresses , so it should be the "same" vm in vCloud , and in case of issues , we would just delete it from vCloud and use the original one in vSphere . Why I am asking ... is that there are licenses bound to hardware uuid / mac addresses , so it is important that we keep the identical vm in any situation.

Any advises on this ?

Thank you.

Edge Gateway currently cannot do SSL termination. So your LB is just doing passthrough and therefore X-Forwarded-For header cannot be added. This will be possible with NSX Edges when they will fully integrate with vCloud Director. New Networking Services Open Door to the Hybrid Cloud | VMware vCloud Blog - VMware Blogs

I have described here how NSX Edge deployed outside of vCloud Director can be utilized for load balancing VCD Cells with the X-Forwarded-For option: https://fojta.wordpress.com/2014/07/08/load-balancing-vcloud-director-cells-with-nsx-edge-gateway/

Tomas

I discovered that reactions have nothing to do with our performance problems. In this particular case, reactions are triggered for hosts that aren't upgraded to the last vCloud agent version.

We have discovered that when the problem occurs backend activity pools remain in a waiting state much more than usual. In the screenshot you can see the threads tree while performing a reconfigure for 6 VMs (operations belonging to a vApp poweron operation)

Image may be NSFW.
Clik here to view.

Looking at pool-161 closely we see::

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.3:2015-07-15 11:05:42,926 | INFO    | backend-activity-pool-161 | VC20VirtualServer | Invoking reconfigure for vm with valref [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=vm-561093]. ConfigSpec change version = 2015-07-15T09:02:47.2151Z. | vcd=0530717f-2e1c-418f-82f4-fce336295513,task=6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:76f358a3-2e05-49e7-8472-bd030cfdec13

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.5:2015-07-15 11:05:43,449 | INFO    | backend-activity-pool-161 | TaskManager | waitForCompletion invoked on task [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=task-2258528] | vcd=0530717f-2e1c-418f-82f4-fce336295513,task=6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:76f358a3-2e05-49e7-8472-bd030cfdec13

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.5:2015-07-15 11:05:43,449 | DEBUG    | backend-activity-pool-161 | TaskManager | Created wait handle for task [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=task-2258528] | vcd=0530717f-2e1c-418f-82f4-fce336295513,task=6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:76f358a3-2e05-49e7-8472-bd030cfdec13

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.5:2015-07-15 11:05:43,449 | DEBUG    | backend-activity-pool-161 | TaskManager | Task [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=task-2258528] will wait for 900 seconds to receive completion notification before timing out | vcd=0530717f-2e1c-418f-82f4-fce336295513,task=6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:6fdced51-1a9c-4fc6-9f90-c795917ec111 activity=urn:uuid:76f358a3-2e05-49e7-8472-bd030cfdec13

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.4:2015-07-15 11:06:31,603 | INFO    | e80575a-6db7-4760-a63a-b28bdf350c6dListener (9381) | TaskManager | inserted task in db. Task = [vcId = 1e80575a-6db7-4760-a63a-b28bdf350c6d, moref = task-2258528, status = 3, completionDate = Wed Jul 15 11:05:49 CEST 2015, age = 0] |

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.4:2015-07-15 11:07:14,412 | DEBUG    | e80575a-6db7-4760-a63a-b28bdf350c6dListener (9381) | MessageBusAdapter | Publishing TaskUpdate [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=task-2258528] |

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.4:2015-07-15 11:07:14,414 | DEBUG    | ActiveMQ Session Task    | MessageBusAdapter | Received TaskUpdate [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=task-2258528] (JMS message ID ID:nex-vcloud-06-v.ucs.datacenter.nexica.com-56690-1436746707406-2:3:1:1:37686) |

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log.4:2015-07-15 11:07:14,415 | DEBUG    | ActiveMQ Session Task    | TaskManager | Handling completion update from MessageBusAdapter for task [vcId=1e80575a-6db7-4760-a63a-b28bdf350c6d, moref=task-2258528] with state SUCCESS |

Looking at vCenter we know for sure the task lasted for a few seconds only:

Looking at vCenter we know that the task lasted for a few seconds:

But the thread backend-activity-pool-161 doesn’t notice about the task update until 11:07:14, more than 80 seconds later…

The question would be, what can be the cause of the delay between the effective task ending and the “MessageBusAdapter | Publishing TaskUpdate” that occurs at 11:07:14. Some problems with activeMQ maybe?

I'm running a resource pod with five ESXi 5.5.0 (2718055) hosts. The PvDC uses a VXLAN network pool with an underlying dedicated VLAN, MTU on VMKNics and physical switches is 9000.

vShield Manager shows "Ready" for the cluster's network preparation. vCloud Director (5.6.4) shows a green check mark for four of the hosts but not for the most recently added one.

I already unprepared and reprepared the host in vCloud Director.

Any ideas what's missing here?

Are you using VXLAN based Network Pools or vCNI pools?

If you are using VXLAN Exclusively (no vCNI pools exist), then I think this is fine+normal+expected.

vCNI is only going to happen when required, which is to have a vCNI network pool existing in vCD and then Prepare the ESXi host.  This installs the vCD agent which is required for vCNI to function.  If vCNI is not required, then you don't need the agent installed (since vShield handles the VXLAN bits).

Hi,

Sorry about the delay but I have been away.

I have saved a deployed vAPP with 16 vm's to the catalog in VCD 5.12.

The original vAPP is still deployed and running and by consolidating functions of several vm's we concluded we could remove 2 virtual machines from the deployed and running environment that no longer served any purpose. I would like to remove the same 2 virtual machines from the vAPP that is saved to the catalog but there is no option (that I could find in the web interface) that would provide me with a means to remove these virtual machines from the saved to the catalog vAPP.

I will have a look at the API as you suggest but would have thought it would be possible through the web interface - apparently not.

Thanks,

Henry

You can't do that to my knowledge.  You'll need to deploy a vApp from the catalog item, remove the offending vm's, then save it back into the catalog.  It really only takes a few minutes if you have linked clones enabled.  Regardless of your storage configuration I don't think there is any way to modify the catalog item once its been created.

Paul

this is correct.  you cannot modify the contents of a vApp in a catalog directly in the way you want.

In vCD 5.1.x, you would deploy a copy ... make appropriate changes (delete VMs) ... capture to catalog under a different name.

In vCD 5.5.x, there is a second path.

you would deploy a copy ... make appropriate changes (delete VMs) ... capture to catalog under the same name, replacing the previous instance (this increases the version displayed by 1).

Thanks. Not exactly the answer I wanted to hear, but at least I know now.

Have a great day!

Hi all,

Im deploying vcloud director and the installation was correct.

But when Im trying to access by https://my_ip/ In this moment the service of the cell in vCD stops and the webserver response:

This webpage is not available

ERR_CONNECTION_REFUSED

Googleing I found a couple of solutions but does not work:

• In previous versions the problem was the Oracle Listener but not in this case because has been possible to connect to the Oracle server by telnet.
• Adding the DNS server in the /etc/hosts to get a complete name resolution but does not work.

I copy also the log of the

Any more ideas?

thx all

Firstly,do check for the browser compatibility .Does the cell service shows as started in cell logs ? When we tail cell and debug logs during browser access what are we seeing in the logs?

Hi,

I am unable to quiesce one of my Vcd Cells. Its seems to be waiting for 1 job to finish for almost 24 hours now

Job count = 1

Is Active = false

On VCD console i can see the following activity running (started 24 hours ago)

Caching organization system

Type: Organization.

Any clues about what this is supposed to mean?

Hi Sreec

Thnaks for your answer the last Friday:

- I checked the browser compatibility and is OK.

- I copy the tail over the cell.log. When I try to connect viw browser the log shows....

Bootstrap application: start

All required system properties are present

All required local configuration properties are present

Successfully bound network port: 80 on host address: MY_VCD_IP

Successfully bound network port: 443 on host address: MY_VCD_IP

Successfully bound network port: 8999 on host address: MY_VCD_IP

All required local configuration properties are present

Successfully initialized system cryptography

Successfully configured HTTP SSL Connector from certificate store: /opt/vmware/vcloud-director/etc/certificates

Current locale "en" verified successfully.

Bootstrap application: complete

Application startup event: Waiting for subsystem 'com.vmware.vcloud.common.core'

Application startup event: Waiting for subsystem 'com.vmware.vcloud.cloud-proxy-server'

Application startup event: Waiting for subsystem 'com.vmware.vcloud.consoleproxy'

Application startup event: Subsystem 'com.vmware.vcloud.common.core' startup initiated.

Application startup begins: Subsystem 'com.vmware.vcloud.common.core' at 7/27/15 10:35 AM

Successfully connected to database: jdbc:oracle:thin:@MY_ORACLE_IP:1521/DOMAIN

Successfully initialized system cryptography

Application startup event: Subsystem 'com.vmware.vcloud.cloud-proxy-server' startup initiated.

Application startup begins: Subsystem 'com.vmware.vcloud.cloud-proxy-server' at 7/27/15 10:35 AM

Application Initialization: 'com.vmware.vcloud.cloud-proxy-server' 33% complete. Subsystem 'com.vmware.vcloud.cloud-proxy-server' started

Uncaught Exception. Originating thread: Thread[bootstrap-application-1,5,main]. Message: Error creating bean with name 'persistenceExceptionTranslationPostProcessor' defined in URL [bundleentry://146.fwk131974647/META-INF/spring/common-ut

il-context.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'globalSessionFactory' defined in URL [bundleentry://146.fwk131974647/META-INF/spri

ng/common-persist-context.xml]: Cannot create inner bean 'com.vmware.vcloud.common.persist.MetadataEntityDeletionInterceptor#660811e7' of type [com.vmware.vcloud.common.persist.MetadataEntityDeletionInterceptor] while setting bean propert

y 'entityInterceptor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vcloud.common.persist.MetadataEntityDeletionInterceptor#660811e7' defined in URL [bundleentry://

146.fwk131974647/META-INF/spring/common-persist-context.xml]: Cannot resolve reference to bean 'cellLivenessStatusService' while setting bean property 'cellLivenessStatusService'; nested exception is org.springframework.beans.factory.Bean

CreationException: Error creating bean with name 'cellLivenessStatusService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.q

uartz.Scheduler com.vmware.vcloud.common.persist.impl.CellLivenessStatusServiceImpl.scheduler; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jobScheduler' defined in URL [bundl

eentry://146.fwk131974647/META-INF/spring/common-concurrent-bundle-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static org.quartz.Sc

heduler com.vmware.vcloud.scheduler.impl.QuartzSchedulerFactory.createScheduler(com.vmware.vcloud.common.configuration.ConfigurationService) throws org.quartz.SchedulerException] threw exception; nested exception is java.lang.IllegalState

Exception: Cannot run without an instance id.

- And I copy some errors of the vcloud-container-debug.log.

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'globalConversationFactory' defined in URL [bundleentry://146.fwk1145006868/META-INF/spring/common-persist-context.xml]: Cannot resolve refe

rence to bean 'globalSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'globalSessionFactory': FactoryBean w

hich is currently in creation returned null from getObject

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:329)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:107)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1360)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1118)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:323)

        ... 73 more

Caused by: org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'globalSessionFactory': FactoryBean which is currently in creation returned null from getObject

        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:156)

        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:109)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1442)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:248)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:323)

Yes, I'm using VXLAN exclusively. I wonder why the other hosts have the green check mark but won't dig into this deeper if not having it is expected. Thanks!

Hi,

    Error message is certainly informative.What version of VCD and Oracle DB we are using in this set-up? You also confirmed DNS is configured properly,but we are still getting

nested exception is java.lang.IllegalState

Exception: Cannot run without an instance id.

Are we 100% sure that DNS is set-up properly in VCD,Machine which is hosting DB,VC and respective Host which are managed in this set-up