AppSuite:Spreadsheet:Installation:Mode-Server-Distributed

From Open-Xchange
Revision as of 10:59, 2 October 2015 by Andreas.schluens (talk | contribs) (update images)

Tutorial : OX Spreadsheet deployment - Mode 'Server Distributed'

General Information

This article describes what you need for an OX Spreadsheet setup - in Mode 'Server Distributed'.

It contains everything you need to:

  • Understand the design of the OX Spreadsheet setup including additional services
  • Install the whole system based on the relevant articles
  • Find pointers to the next steps of integration


System Design

AppSuite-Spreadsheet-Installation-Mode-Server-Distributed-Physical-Architecture.jpg AppSuite-Spreadsheet-Installation-Mode-Server-Distributed-Logical-Architecture.jpg

The system is designed to provide maximum functionality and availability with a minimum of necessary hardware. If the services on one OX server fail, this is transparently handled by others within the same cluster.

Core Components

  • Nx OX AppSuite (HW recommendation: 16GB RAM / 4 cores) running as OX Spreadsheet Client
    (follow the instructions regarding Nature 'OX-Backend')
  • 1x OX Spreadsheet Server (HW recommendation: 8GB RAM / 2 cores)
    (follow the instructions regarding Nature 'CE-Server')
  • Nx OX Spreadsheet Worker (HW recommendation: 24GB RAM / 4 cores)
    (follow the instructions regarding Nature 'CE-Worker')

Overview Installation Steps

To deploy the described OX Spreadsheet setup, the following steps need to be done.

Mandatory Installation Steps

  • Install and configure OX Appsuite - Nature 'OX-Backend'
  • Install and configure OX Spreadsheet Server - Nature 'CE-Server'
  • Install and configure OX Spreadsheet Worker - Nature 'CE-Worker'

Configuration Steps

  • Configure OX Appsuite - Nature 'OX-Backend'
  • Configure OX Spreadsheet Server - Nature 'CE-Server'
  • Configure OX Spreadsheet Worker - Nature 'CE-Worker'

Start & Stop

  • Start/Stop OX Appsuite - Nature 'OX-Backend'
  • Start/Stop OX Spreadsheet Server - Nature 'CE-Server'
  • Start/Stop OX Spreadsheet Worker - Nature 'CE-Worker'

Mandatory Installation Steps - Instructions & Recommendations

Install and configure OX Appsuite - Nature 'OX-Bckend'

The following steps need to be done to get OX Spreadsheet Client up and running:

Install and configure OX Spreadsheet Server - Nature 'CE-Server'

The following steps need to be done to get OX Spreadsheet Server HA up and running:

Install and configure OX Spreadsheet Worker - Nature 'CE-Worker'

The following steps need to be done to get OX Spreadsheet Worker HA up and running:

Configuration Steps - Instructions & Recommendations

Configure OX Appsuite - Nature 'OX-Backend'

After installation the OX Spreadsheet functionality is disabled by default.
To enable OX Spreadsheet Application the associated permission has to be set.
The default setting for all users is changed in the file/opt/open-xchange/etc/permissions.properties. 

# Note:
# This list is a comma seperated list of different permissions.
# Add or remove the permission you want to change - leave other permissions unchanged.

permissions=..,spreadsheet,..

By enabling the OX Spreadsheet Application the corresponding OX Spreadsheet Portal was enabled also.
If that is not intended you can disable the OX Spreadsheet Portal within the same configuration file/opt/open-xchange/etc/permissions.properties: 

# Note:
# This list is a comma seperated list of different permissions.
# Add or remove the permission you want to change - leave other permissions unchanged.

permissions=..,spreadsheet,spreadsheetportaldisabled,..

Enable client mode at this node inside configuration file/opt/open-xchange/etc/calcengine.properties: 

# -----------------------------------------------------------------------------
# define the mode the calc engine will use at runtime
...
# - jms
#       all involved processes will look for a JMS server to communicate together.
...
#

calcengine.mode=jms

The mode 'Server Distributed' was designed for scalability.
This require a different handling of the load balancer strategy.
Configure inside the configuration file/opt/open-xchange/etc/calcengine.propertiesthe following: 

# -----------------------------------------------------------------------------
# define where and how the load balancer is running.
#
# - server-dynamic-router
#           Its running at server side using the dynamic-router pattern.
#           As dynamic-router only header of JMS messages are interpreted and used.
#           Doing so not the whole JMS message needs to be submitted to the load balancer process.
#
# - client-embedded
#           Its running embedded at client side. So it's not a dedicated process at server side.
#           At least it's nothing more then a lookup into the registry of worker processes
#           and a mapping of request to worker queues.
#           This mode is very fast and scalable - but provides less feature then all other modes.
#           E.g. re-queuing of messages in  case worker is restarting is not provided.
#
# default = server-dynamic-router
#
calcengine.load-balancer.type=client-embedded

The OX Spreadsheet Client needs to know where the OX Spreadsheet Server is running.
But the client is not connected to the server directly.
As communication channel between both nodes JMS is used.

The OX Spreadsheet Server acts as a JMS Server also.
Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ox-backend/jms-server.properties
to generate a suitable configuration at/opt/open-xchange/etc/documents/jms-server.properties
and replace all place holder following the schema%%[...]%%.

Note: All place holder values must be in sync with all other configuration files within the same setup. 

# =============================================================================
# NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
#        ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

# -----------------------------------------------------------------------------
# define the host (IP/DNS) where the jms server is running and we have to
# be connected with
#
# default=127.0.0.1
#
jms.server.host=%%[CE_SERVER_IP]%%

# -----------------------------------------------------------------------------
# define the port where the server should listen for incoming requests
# or a client can be connected with those server.
#
# default=61616
#
#jms.server.port=61616

The OX Spreadsheet Server acts as a Cache Server also.
Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ox-backend/cache-server.properties
to generate a suitable configuration at/opt/open-xchange/etc/documents/cache-server.properties
and replace all place holder following the schema%%[...]%%.

Note: All place holder values must be in sync with all other configuration files within the same setup. 

# =============================================================================
# NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
#        ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

# -----------------------------------------------------------------------------
# define the host (IP/DNS) where the jms server is running and we have to
# be connected with
#
# default=127.0.0.1
#
cache.server.host=%%[CE_SERVER_IP]%%

# -----------------------------------------------------------------------------
# define the port where the server should listen for incoming requests
# or a client can be connected with those server.
#
# default=7000
#
#cache.server.port=7000

# -----------------------------------------------------------------------------
# unique ID for this server instance.
# Can be used to separate multiple server instances on the same machine.
#
cache.server.id=%%[CE_SERVER_ID]%%

# -----------------------------------------------------------------------------
# password to join those cache server.
# Can be empty.
#
cache.server.password=%%[CE_SERVER_PASSWORD]%%

Configure OX Spreadsheet Server - Nature 'CE-Server'

The OX Spreadsheet Server host:

  • 1x JMS Server for communication
  • 1x Cache Server for caching data


The mode 'Server Distributed' was designed for scalability.
This require a different handling of the load balancer strategy.
Configure inside the configuration file/opt/open-xchange/etc/calcengine.propertiesthe following: 

# -----------------------------------------------------------------------------
# define the mode the calc engine will use at runtime
#
# - jms
#       calc engine is a central JMS server to communicate with
#       all configured worker processes even clustered on different machines.
#
calcengine.mode=jms

# -----------------------------------------------------------------------------
# define where and how the load balancer is running.
#
# - server-dynamic-router
#           Its running at server side using the dynamic-router pattern.
#           As dynamic-router only header of JMS messages are interpreted and used.
#           Doing so not the whole JMS message needs to be submitted to the load balancer process.
#
# - client-embedded
#           Its running embedded at client side. So it's not a dedicated process at server side.
#           At least it's nothing more then a lookup into the registry of worker processes
#           and a mapping of request to worker queues.
#           This mode is very fast and scalable - but provides less feature then all other modes.
#           E.g. re-queuing of messages in  case worker is restarting is not provided.
#
# default = server-dynamic-router
#
calcengine.load-balancer.type=client-embedded

Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ce-server/jms-server.properties
to generate a suitable configuration at/opt/open-xchange/etc/documents/jms-server.properties
and replace all place holder following the schema%%[...]%%. 

# =============================================================================
# NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
#        ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

# -----------------------------------------------------------------------------
# define the network interface where the server should listen for incoming requests.
#
# Possible values are:
#   - 0.0.0.0   to allow all (internal and external) connections
#   - 127.0.0.1 to allow local connections only
#
# default=127.0.0.1
#
jms.server.interface=0.0.0.0

# -----------------------------------------------------------------------------
# define the port where the server should listen for incoming requests
# or a client can be connected with those server.
#
# default=61616
#
#jms.server.port=61616

# -----------------------------------------------------------------------------
# define directory for storing persistent data
#
# Will be created automatically if it does not exists.
# If it's not defined persistence will be disabled.
#
# Note: Thise feature can be disabled by setting an empty path only !
#
#jms.server.persistence.dir=${sys:java.io.tmpdir}/ox-jms-embedded-server/persistence

# -----------------------------------------------------------------------------
# frequency where the JMS Garbage Collector (GC) should run
#
# unit=[ms]
# default=300000 (5min)
#
#jms.server.gc.frequency=300000

# -----------------------------------------------------------------------------
# the time a JMS resource (queue/topic) has to be inactive before it can
# will be removed by the GC
#
# unit=[ms]
# default=60000 (1min)
#
#jms.server.gc.inactive-timeout-before-gc=60000

Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ce-server/cache-server.properties
to generate a suitable configuration at/opt/open-xchange/etc/documents/cache-server.properties
and replace all place holder following the schema%%[...]%%. 

# =============================================================================
# NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
#        ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

# -----------------------------------------------------------------------------
# define the network interface where the server should listen for incoming requests.
#
# Possible values are:
#   - <IP_RANGE> to allow all (internal and external) connections
#   - 127.0.0.1  to allow local connections only
#
# <IP_RANGE> example:
#           IP of local machine is "192.168.0.1".
#           A might suitable IP range can be "192.168.0.*"
#
# default=127.0.0.1
#
cache.server.interface=%%[CE_CLIENTS_IP_RANGE]%%

# -----------------------------------------------------------------------------
# define the port where the server should listen for incoming requests
# or a client can be connected with those server.
#
# default=7000
#
#cache.server.port=7000

# -----------------------------------------------------------------------------
# unique ID for this server instance.
# Can be used to separate multiple server instances on the same machine.
#
# default = 'default'
#
cache.server.id=%%[CE_SERVER_ID]%%

# -----------------------------------------------------------------------------
# password to join those cache server.
# Can be empty.
#
# default = 'xxx'
#
cache.server.password=%%[CE_SERVER_PASSWORD]%%

# -----------------------------------------------------------------------------
# define directory for storing persistent data
#
# Will be created automatically if it does not exists.
# If it's not defined persistence will be disabled.
#
# Note: To disable this feature define an empty path.
#
# default = '${sys:java.io.tmpdir}/ox-cache-server/${cache.server.id}/persistence'
#
#cache.server.persistence.dir=${sys:java.io.tmpdir}/ox-cache-server/${cache.server.id}/persistence

The OX Spreadsheet Server - Mode 'Server Distributed' do not run OX Spreadsheet Worker locally.
They have to be disabled. This and other apsects of this setup can be configured within the file/opt/open-xchange/etc/documents/cluster-node.xml:
Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ce-server/cluster-node.xmlto create those file. 

<!-- ==========================================================================

NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
       ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

  -->

<cluster>

    <node>

        <cache>
            <!-- ==============================================================
                 directory where states about running apps can be made persistent.
              
                 Note: To disable this feature define an empty path.
                 
                 default = ${sys:java.io.tmpdir}/ox-cluster-manager/cache
              -->
            <!--
            <dir>${sys:java.io.tmpdir}/ox-cluster-manager/cache</dir>
              -->
        </cache>

        <watchdog>
            <!-- ==============================================================
                 poll time to detect dead processes in [s]
              
                 unit    = [s]
                 default = 10s
              -->
            <!-- 
            <poll-time>10</poll-time>
              -->
              
            <!-- ==============================================================
                 timeout for starting one app in [s]
              
                 unit    = [s]
                 default = 60s
              -->
            <!--
            <appstart-timeout>60</appstart-timeout>
              -->
        </watchdog>
        
        <apps merge-operation="explicit">

            <!-- ==============================================================
                 enable and configure the central jms server ...
              -->
            <app id="jms-server">
                <!-- ==========================================================
                     define the memory this process can consume ...
                  
                     - the 'min' value is used to reserver memory at startup
                     - the 'max' value is used to define the maximum range which can be consumed
                  
                     default for min   = 1024
                     default for max   = 2048
                  -->
                <!-- 
                <memory>
                    <min>1024</min>
                    <max>2048</max>
                </memory>
                 -->
            </app>
            
            <!-- ==============================================================
                 enable and configure the central cache server ...
              -->
            <app id="cache-server">
                <!-- ==========================================================
                     define the memory this process can consume ...
                  
                     - the 'min' value is used to reserver memory at startup
                     - the 'max' value is used to define the maximum range which can be consumed
                  
                     default for min   = 1024
                     default for max   = 2048
                  -->
                <!-- 
                <memory>
                    <min>128</min>
                    <max>256</max>
                </memory>
                 -->
            </app>
            
            <!-- ==============================================================
                 no dedicated calc engine server in this mode ...
              -->
            
            <!-- ==============================================================
                 no calcengine worker processes on this node ...
              -->

        </apps>
    </node>
</cluster>

Configure OX Spreadsheet Worker - Nature 'CE-Worker'

The OX Spreadsheet Worker host:

  • Nx Worker processes connected


The OX Spreadsheet Worker node host all worker processes where the real user documents are handled.
So e.g. the persistence of such documents (usable for an automatic document recovery in case a worker was restarted) has to be configured at this node.
Inside the configuration file/opt/open-xchange/etc/calcengine.propertiesdefine the following: 

# -----------------------------------------------------------------------------
# define the mode the calc engine will use at runtime
#
# - jms
#       calc engine is a central JMS server to communicate with
#       all configured worker processes even clustered on different machines.
#
calcengine.mode=jms

# -----------------------------------------------------------------------------
# path where worker can cache there documents to support auto-restore in case
# a worker crashed and will be reactivated.
#
# If this path is empty - auto-restore will be disabled.
#
# This value can use special variable $(TEMP) to place the cache
# within the temporary directory. Other variables are not supported.
#
# Note: This feature can be disabled by define an empty path only !
#
calcengine.restore-documents.path=${sys:java.io.tmpdir}/ox-ce-doc-cache

The OX Spreadsheet Worker needs to know where the JMS Server is running.
JMS acts as a communication channel between the OX Spreadsheet Worker and Server.
Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ce-worker/jms-server.properties
to generate a suitable configuration at/opt/open-xchange/etc/documents/jms-server.properties
and replace all place holder following the schema%%[...]%%. 

# =============================================================================
# NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
#        ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

# -----------------------------------------------------------------------------
# define the host (IP/DNS) where the jms server is running and we have to
# be connected with
#
# default=127.0.0.1
#
jms.server.host=%%[CE_SERVER_IP]%%

# -----------------------------------------------------------------------------
# define the port where the server should listen for incoming requests
# or a client can be connected with those server.
#
# default=61616
#
#jms.server.port=61616

The OX Spreadsheet Worker needs to register itself at OX Spreadsheet Server.
The OX Spreadsheet Server host also a registry (cache) server.
Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ce-worker/cache-server.properties
to generate a suitable configuration at/opt/open-xchange/etc/documents/cache-server.properties
and replace all place holder following the schema%%[...]%%. 

# =============================================================================
# NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
#        ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

# -----------------------------------------------------------------------------
# define the host (IP/DNS) where the jms server is running and we have to
# be connected with
#
# default=127.0.0.1
#
cache.server.host=%%[CE_SERVER_IP]%%

# -----------------------------------------------------------------------------
# define the port where the server should listen for incoming requests
# or a client can be connected with those server.
#
# default=7000
#
#cache.server.port=7000

# -----------------------------------------------------------------------------
# unique ID for this server instance.
# Can be used to separate multiple server instances on the same machine.
#
cache.server.id=%%[CE_SERVER_ID]%%

# -----------------------------------------------------------------------------
# password to join those cache server.
# Can be empty.
#
cache.server.password=%%[CE_SERVER_PASSWORD]%%

The OX Spreadsheet Worker do not run other services locally.

They have to be disabled. This and other apsects of this setup can be configured within the file/opt/open-xchange/etc/documents/cluster-node.xml:
Use the template/opt/open-xchange/etc-templates/documents/mode-server-distributed/nature-ce-worker/cluster-node.xmlto create those file. 

<!-- ==========================================================================

NOTE : DO NOT USE DEFAULT VALUES WITHIN THIS SCRIPT AS THEY COME FROM THE CODE.
       ACTIVATE AND CHANGE VALUES IF NEEDED ONLY.

  -->

<cluster>

    <node>

        <cache>
            <!-- ==============================================================
                 directory where states about running apps can be made persistent.
              
                 Note: To disable this feature define an empty path.
                 
                 default = ${sys:java.io.tmpdir}/ox-cluster-manager/cache
              -->
            <!--
            <dir>${sys:java.io.tmpdir}/ox-cluster-manager/cache</dir>
              -->
        </cache>

        <watchdog>
            <!-- ==============================================================
                 poll time to detect dead processes in [s]
              
                 unit    = [s]
                 default = 10s
              -->
            <!-- 
            <poll-time>10</poll-time>
              -->
              
            <!-- ==============================================================
                 timeout for starting one app in [s]
              
                 unit    = [s]
                 default = 60s
              -->
            <!--
            <appstart-timeout>60</appstart-timeout>
              -->
        </watchdog>
        
        <apps merge-operation="explicit">

            <!-- ==============================================================
                 no jms server on this node ...
              -->

            <!-- ==============================================================
                 no cache server on this node ...
              -->
            
            <!-- ==============================================================
                 no calc engine server on this node ...
              -->

            <!-- ==============================================================
                 enable and configure all calcengine worker processes here ...
              -->
            <app id="calcengine-worker">
            
                <!-- ==========================================================
                     define how and when those calcengine worker has to be started.

                     The following modes are available:

                     auto      : the configured minimum amount of processes is started automatically
                                 if the OX backend will be started.
                     on-demand : the configured minimum amount of processes is started at runtime
                                 if the first spreadsheet document is requested.
                
                     default = on-demand
                  -->
                <start-mode>auto</start-mode>

                <!-- ==========================================================
                     define the minimum amount of worker processes to be started ...
                     
                     ... immediately on start up of OX backend in mode 'auto'
                     ... lazy on first spreadsheet request in mode 'on-demand'.
                  
                     range   = [0..max-instances]
                     default = 2
                  -->
                <min-instances>10</min-instances>
                
                <!-- ==========================================================
                     define the maximum amount of worker processes to be used at runtime ...
                  
                     range   = [min-instances...]
                     default = 2
                  -->
                <max-instances>10</max-instances>

                <!-- ==========================================================
                     define the memory one worker can consume ...
                  
                     - the 'min' value is used to reserver memory at startup
                     - the 'max' value is used to define the maximum range which can be consumed
                     - the 'limit' value is used to KILL the process
                       because it exceeds the maximum amount or memory
                  
                     default for min   = 1024
                     default for max   = 2048
                     default for limit = 2100
                  -->
                <!-- 
                <memory>
                    <min>1024</min>
                    <max>2048</max>
                    <limit>2100</limit>
                </memory>
                 -->

            </app>

        </apps>
    </node>
</cluster>

Start & Stop - Instructions & Recommendations

Note: In general the order of starting all nodes (OX Appsuite, OX Spreadsheet Server and OX Spreadsheet Worker) is not important. An automatic reconnect feature guarantee a stable environment at runtime.
On the other side it's more ease to analyze and locate errors inside the deployment if both nodes are started in a special order :
OX Spreadsheet Server firstAll OX Spreadsheet Worker thenOX Appsuite at last

Start/Stop OX Spreadsheet Server - Nature 'CE-Server'

The OX Spreadsheet Server can be controlled by the script/etc/init.d/open-xchange-cluster-management. The OX Backend is not installed and has not to be started on this node. 

# start OX Spreadsheet Server
/etc/init.d/open-xchange-cluster-management start

...

# check status of OX Spreadsheet Server
/etc/init.d/open-xchange-cluster-management status

...

# stop OX Spreadsheet Server
/etc/init.d/open-xchange-cluster-management stop

Start/Stop OX Spreadsheet Worker - Nature 'CE-Worker'

The OX Spreadsheet Worker can be controlled by the script/etc/init.d/open-xchange-cluster-management. The OX Backend is not installed and has not to be started on this node. 

# start OX Spreadsheet Worker
/etc/init.d/open-xchange-cluster-management start

...

# check status of OX Spreadsheet Worker
/etc/init.d/open-xchange-cluster-management status

...

# stop OX Spreadsheet Worker
/etc/init.d/open-xchange-cluster-management stop

Start/Stop OX Appsuite - Nature 'OX-Backend'

The OX Spreadsheet Client is running within the OX Backend process. So it's enough to control those backend service by calling the script/etc/init.d/open-xchange. 

# start OX Backend
/etc/init.d/open-xchange start

...

# check status of OX Backend
/etc/init.d/open-xchange status

...

# stop OX Backend
/etc/init.d/open-xchange stop
Retrieved from "https://wiki.open-xchange.com/wiki/index.php?title=AppSuite:Spreadsheet:Installation:Mode-Server-Distributed&oldid=20636"