I'll try to test it and let you know if it worked.

Thank you.

What you describe only occurs in case of a crash or network partition, and in that case indeed we don't support automatic cleanup, and the cleanup on next startup of a Nuxeo node will fail as you describe if the session_id is reused externally.

We don't have a uniquely identifying Nuxeo node instance id at the moment, although that's something we want to add (in order to facilitate clustering and not have issues like the one you describe).

We'll look for any error in the logs during shutdown and let you know what we find.

Thanks for your help.

I'm afraid a correct shut down does not delete the entry from cluster_nodes.

I created a trigger in cluster_nodes to log every delete. After shutting down the instance no delete operation was recorded. When I started it, the delete operation took place and the log appeared. No error or warning appears in Nuxeo server.log.

After downloading the source code I look for references to cluster_nodes and only find the txt files in nuxeo-core-storage-sql. Where could I find the cleanup code?

Additionaly, we are also going to modify the cleanup of cluster_invals. Since it also depends on the session, a SQLServer failover make the cleanup fail until the Nuxeo nodes are restarted (a failover will force the session to reconnect and get a new session_id).

Thanks in advance.

If you look at sqlserver.sql.txt you'll see a section:

#CATEGORY: removeClusterNode

DELETE FROM [cluster_nodes] WHERE nodeid = @@SPID;

# Remove orphan invalidations
DELETE FROM [cluster_invals] WHERE [nodeid] IN (
  SELECT DISTINCT [cluster_invals].[nodeid]
    FROM [cluster_invals] LEFT JOIN [cluster_nodes] ON [cluster_invals].[nodeid] = [cluster_nodes].[nodeid]
    WHERE [cluster_nodes].[nodeid] IS NULL
)

which is the code executed at shutdown. If you uncomment this section in lib/log4j.xml:

<!-- Uncomment to debug SQL statements -->
<!--
<category name="org.nuxeo.ecm.core.storage.sql.jdbc">
  <priority value="TRACE" />
</category>
-->

then you'll see all SQL statements executed in the log/server.log file. At shutdown you should see among other things:

TRACE [JDBCLogger] (2) SQL: DELETE FROM [cluster_nodes] WHERE nodeid = @@SPID;
TRACE [JDBCLogger] (2) SQL: DELETE FROM [cluster_invals] WHERE [nodeid] IN (
  SELECT DISTINCT [cluster_invals].[nodeid]
    FROM [cluster_invals] LEFT JOIN [cluster_nodes] ON [cluster_invals].[nodeid] = [cluster_nodes].[nodeid]
    WHERE [cluster_nodes].[nodeid] IS NULL

This is the modified version log at startup: 2014-07-18 13:00:47,100 TRACE [localhost-startStop-1] org.nuxeo.ecm.core.storage.sql.jdbc.JDBCLogger SQL: DELETE FROM [cluster_nodes] where hostname =

(select host_name collate DATABASE_DEFAULT FROM sys.dm_exec_sessions S WHERE S.session_id = @@SPID)

But there is no DELETE at shutdown. Actually there is no much at shutdown in the log, only two rows (log is set at WARN for most components): 2014-07-18 13:01:44,352 INFO [localhost-startStop-2] [org.nuxeo.ecm.core.storage.sql.jdbc.QueryMakerServiceImpl] Unregistering QueryMaker 'NXQL': org.nuxeo.ecm.core.storage.sql.jdbc.NXQLQueryMaker 2014-07-18 13:01:45,615 INFO [localhost-startStop-2] [org.nuxeo.ecm.core.storage.sql.jdbc.QueryMakerServiceImpl] Unregistering QueryMaker 'CMISQL': org.nuxeo.ecm.core.opencmis.impl.server.CMISQLQueryMaker

I'll enable more traces and test it again.

Maybe It's because of the OS (Windows). I'll test with Linux and see what I get.

After enabling the jdbc logs and setting up the cluster, the behaviour is the same (DELETEs occur at startup but not during shutdown).

I'm configuring the cluster according to http://doc.nuxeo.com/display/public/ADMINDOC/Nuxeo+Clustering+Configuration.

Is there any additional step to consider? Could anyone else reproduce the problem?