Sandbox:Middleware Startup/sandbox
Middleware start
Systemd
System V / Upstart
Details explained
6 Evaluate java commandline options
Apply Resource limits
Systemd
Service definition
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service [Unit] After=remote-fs.target After=time-sync.target ypbind.service sendmail.service cyrus.service [Service] User=open-xchange PermissionsStartOnly=true TimeoutStartSec=0 ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u ExecStart=/opt/open-xchange/sbin/open-xchange ExecStop=/opt/open-xchange/sbin/shutdown -w ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d KillMode=process LimitNOFILE=65536 LimitNPROC=65536 [Install] WantedBy=multi-user.target
Drop-in configs
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf # Override and add options in this file # See systemd.exec(5) for other limits [Service] #LimitNPROC=65536 #LimitNOFILE=65536
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.
Note: The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with systemctl daemon-reload. The above is just a simplyfied version to explain the general concept.
System V / Upstart
Service definition
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.
Apply Resource limits
As you can see in the diagrams above one of the differences is that:
- systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram Service startup using systemd) and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
- the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram Service startup using upstart).
Background Infos
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.
Sysctl
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files
$ sysctl -w fs.file-max=100000
To permanently set them append to the main configuration file and reload the settings
$ echo fs.file-max=100000 >> /etc/sysctl.conf $ sysctl -p
More infos can be found via man sysctl
Limits.conf
Allows to restrict resources an a global, group or user level. E.g:
$ cat /etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 1024
From man limits.conf:
Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.
The limits per login are applied via the pam stack. See man pam and man pam_limits for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see upstart, systemd and start-stop-daemon
Ulimit
From man bash
ulimit [-HSTabcdefilmnpqrstuvx [limit]] Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only the maximum number of processes and the maximum number of open file descriptors available to a single user are increased via ulimit. The values are specified in /opt/open-xchange/ox-scriptconf.sh
Systemd
Control Groups
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a modern distribution that already uses systemd as init.
Citing from the kernel cgroup documentation:
1-2. What is cgroup?
cgroup is a mechanism to organize processes hierarchically and distribute system resources along the hierarchy in a controlled and configurable manner.
cgroup is largely composed of two parts - the core and controllers. cgroup core is primarily responsible for hierarchically organizing processes. A cgroup controller is usually responsible for distributing a specific type of system resource along the hierarchy although there are utility controllers which serve purposes other than resource distribution.
cgroups form a tree structure and every process in the system belongs to one and only one cgroup. All threads of a process belong to the same cgroup. On creation, all processes are put in the cgroup that the parent process belongs to at the time. A process can be migrated to another cgroup. Migration of a process doesn't affect already existing descendant processes.
Following certain structural constraints, controllers may be enabled or disabled selectively on a cgroup. All controller behaviors are hierarchical - if a controller is enabled on a cgroup, it affects all processes which belong to the cgroups consisting the inclusive sub-hierarchy of the cgroup. When a controller is enabled on a nested cgroup, it always restricts the resource distribution further. The restrictions set closer to the root in the hierarchy can not be overridden from further away.
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?
5. Controllers
5-1. CPU
The "cpu" controllers regulates distribution of CPU cycles. This controller implements weight and absolute bandwidth limit models for normal scheduling policy and absolute bandwidth allocation model for realtime scheduling policy.
5-2. Memory
The "memory" controller regulates distribution of memory. ... While not completely water-tight, all major memory usages by a given cgroup are tracked so that the total memory consumption can be accounted and controlled to a reasonable extent.
5-3. IO
The "io" controller regulates the distribution of IO resources. This controller implements both weight based and absolute bandwidth or IOPS limit distribution; however, weight based distribution is available only if cfq-iosched is in use and neither scheme is available for blk-mq devices.
The open-xchange service is simply put into the default system.slice without applying further limits.
singlenode$ systemd-cgls ├─1 /sbin/init ├─system.slice │ ├─avahi-daemon.service │ │ ├─501 avahi-daemon: running [singlenode] │ │ └─514 avahi-daemon: chroot helper │ ├─console-kit-daemon.service │ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon │ ├─dbus.service │ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation │ ├─munin-node.service │ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node │ ├─open-xchange.service │ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...
To check all the details use
singlenode:~ # systemctl show system.slice
Limits besides control groups
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via limits.conf or ulimit. Systemd uses setrlimit for this. The options that we set by default are:
* LimitNOFILE * LimitNPROC
which you have already seen when we took a look at the service definition and drop-in configs. Again: The complete documentation wrt. resource limit configurations in drop-in configs can be found at systemd.exec
Verify limits
System V
Get the proper pid of the java process started for the middleware by either checking ps faux
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange 494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600
or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)
and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 65536 65536 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 24254 24254 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Systemd
singlenode:~ # systemctl show open-xchange | grep Limit StartLimitInterval=10000000 StartLimitBurst=5 StartLimitAction=none MemoryLimit=18446744073709551615 LimitCPU=18446744073709551615 LimitFSIZE=18446744073709551615 LimitDATA=18446744073709551615 LimitSTACK=18446744073709551615 LimitCORE=18446744073709551615 LimitRSS=18446744073709551615 LimitNOFILE=65536 LimitAS=18446744073709551615 LimitNPROC=65536 LimitMEMLOCK=65536 LimitLOCKS=18446744073709551615 LimitSIGPENDING=19827 LimitMSGQUEUE=819200 LimitNICE=0 LimitRTPRIO=0 LimitRTTIME=18446744073709551615
Evaluate JVM commandline options
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in /opt/open-xchange/etc/ox-scriptconf.sh
Format < 7.8.4
For version up to 7.8.3 the JVM configuration options were all specified via a single property JAVA_XTRAOPTS which resulted in a rather long configuration string that was hard to read and maintain.
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \ -Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \ -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \ -Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.
Format >= 7.8.4
Beginning from version 7.8.4 the options are split up and sorted into different categories like:
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC" JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow" JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB" JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10" JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false" JAVA_OPTS_SERVER="-server -Djava.awt.headless=true" JAVA_OPTS_OTHER="" #JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \ -XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"
which is much easier to read and maintain. Furthermore we added a useful JAVA_OPTS_DEBUG configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.
Upgrade procedure
During the upgrade 7.8.3-revX -> 7.8.4-revY existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.
Encoding detection during JVM startup
Background Infos
From man 7 locale
DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.
The format to specify a system locale is: language[_territory][.codeset][@modifier].
Where all of
- en_US.UTF-8
- en_US.utf8
- en_US.utf-8
- en_US.UTF_8
will result in a valid UTF-8 charmap for the english language with locale categories configured as
LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=
See for more infos: locale(1), locale(7), language codes, country codes and character encodings
JVM startup
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:
CODESET (LC_CTYPE) Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1", or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap". For a list of character encoding names, try "locale -m", cf. locale(1).
Furthermore this influences the defaultCharset selection of the JVM.
/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}
Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every service start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).
See the documentation of your linux distribution on how to configure a proper unicode locale.
Query which file encoding was detected by the JVM during startup
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like visualvm, query our Jolokia JMX bridge via http or just use the commandline like
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'
{key=file.encoding, value=UTF-8}
Locale passing from client to server via SSH
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.
grep AcceptEnv /etc/ssh/sshd_config AcceptEnv LANG LC_*
For example when i connect to the machine my local german locale is sent to the server
[vagrant@showcase-community-centos-7 ~]$ locale -v LANG=de_DE.UTF-8 LC_CTYPE="de_DE.UTF-8" LC_NUMERIC=de_DE.UTF-8 LC_TIME=de_DE.UTF-8 LC_COLLATE="de_DE.UTF-8" LC_MONETARY=de_DE.UTF-8 LC_MESSAGES="de_DE.UTF-8" LC_PAPER=de_DE.UTF-8 LC_NAME=de_DE.UTF-8 LC_ADDRESS=de_DE.UTF-8 LC_TELEPHONE=de_DE.UTF-8 LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=de_DE.UTF-8 LC_ALL=
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l -bash-4.2$ locale -v LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=
Verify that your system is configured with a proper unicode locale for OX
Assuming you haven't reconfigured the locale of your root user
[root@showcase-community-centos-7 vagrant]# locale charmap UTF-8 [root@showcase-community-centos-7 vagrant]#
Warnings during early start
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.
Limits
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup
singlenode open-xchange # cat open-xchange-console.log /opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted
Locale
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE. WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.
Summary: Open-Xchange middleware on specific distros
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.
Debian 7
- Init
- System V style
- OX Configurable Limits/Defaults
- nofile, nproc
The mentioned limits can be configured via /opt/open-xchange/etc/ox-scriptconf.sh. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider /etc/security/limits.*
RHEL 6 / CentOS 6
- Init
- Upstart, System V compatible
- OX Configurable Limits/Defaults
- nofile, nproc
The mentioned limits can be configured via /opt/open-xchange/etc/ox-scriptconf.sh. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via su ... open-xchange on this distro a user session is opened via su/pam and the default CentOS pam config reads the /etc/security/limits.* configuration by loading the pam stack like:
- /etc/pam.d/su
- -> /etc/pam.d/system-auth
- -> pam_limits.so
- -> /etc/pam.d/system-auth
If NPROC isn't configured for the open-xchange-server it's restricted to 1024 globally by default to prevent accidental fork bombs, see /etc/security/limits.d/90-nproc.conf which can result in severe problems modern multithreaded applications.
RHEL 7 / CentOS 7 / Debian 8 / SLE 12
- Init
- Systemd
- OX Configurable Limits/Defaults
- nofile, nproc
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at /usr/lib/systemd/system/open-xchange.service. The drop-in config to override or extend the default unit file is located at /etc/systemd/system/open-xchange.service.d/limits.conf. Systemd.exec shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.