Context Preprovisioning - Revision history

Dominik.epple at 15:25, 1 August 2017

2017-08-01T15:25:39Z

Dominik.epple: /* Schema pre-generation */

2017-08-01T15:17:02Z

Schema pre-generation

← Older revision		Revision as of 15:17, 1 August 2017
Line 22:		Line 22:

	So with this setting CONTEXTS_PER_SCHEMA=1, you create as much contexts as you require for your final number of contexts to be provisioned. Then you change that setting back to its original value (as per system high level design, usually something between 1000 and 7000).		So with this setting CONTEXTS_PER_SCHEMA=1, you create as much contexts as you require for your final number of contexts to be provisioned. Then you change that setting back to its original value (as per system high level design, usually something between 1000 and 7000).

			Update: as of 7.8.3, there is also a tool called createschema to bootstrap schemas without the aforementioned trick. However, these schemas are only suitable for schema-name based fastmode, so the aforementioned trick is still required for bootstrapping schemas for use with in-memory or automatic mode.

	=== Context Creation in Fast Mode ===		=== Context Creation in Fast Mode ===

Dominik.epple: /* Schema-Name vs Schema-Strategy */

2015-10-01T09:41:04Z

Schema-Name vs Schema-Strategy

← Older revision		Revision as of 09:41, 1 October 2015
Line 49:		Line 49:
	The approach of the --schema-strategy=in-memory is to not require any logic from the caller, but rather do the required book-keeping in memory of OX.		The approach of the --schema-strategy=in-memory is to not require any logic from the caller, but rather do the required book-keeping in memory of OX.

	Both approaches seem to work in our benchmarks with similar achieved throughput. Theoretically the in-memory approach is easier to integrate into existing workflows, but the schema-name approach is much simpler inside of OX ~~ant~~ potentially more robust.		Both approaches seem to work in our benchmarks with similar achieved throughput. Theoretically the in-memory approach is easier to integrate into existing workflows, but the schema-name approach is much simpler inside of OX and potentially more robust.

Dominik.epple at 09:32, 1 October 2015

2015-10-01T09:32:20Z

← Older revision		Revision as of 09:32, 1 October 2015
Line 49:		Line 49:
	The approach of the --schema-strategy=in-memory is to not require any logic from the caller, but rather do the required book-keeping in memory of OX.		The approach of the --schema-strategy=in-memory is to not require any logic from the caller, but rather do the required book-keeping in memory of OX.

	Both approaches seem to work in our benchmarks with similar achieved throughput~~. None of them have undergone extensive real-world usage~~. Theoretically the in-memory approach is easier to integrate into existing workflows, but the schema-name approach is much simpler inside of OX ant potentially more robust.		Both approaches seem to work in our benchmarks with similar achieved throughput. Theoretically the in-memory approach is easier to integrate into existing workflows, but the schema-name approach is much simpler inside of OX ant potentially more robust.

Dennis Sieben: /* Context Preprovisioning */

2015-10-01T09:24:59Z

Context Preprovisioning

← Older revision		Revision as of 09:24, 1 October 2015
Line 1:		Line 1:
	= Context Preprovisioning =		= Context Preprovisioning =

	The standard createcontext call (be it via CLT or RMI or SOAP) is designed to work under all possible cornercases, like concurrent createcontext, createuser, deletecontext, including potential generation and removal of DB schemas and correct counting of schemas on target (DBs, Schemas, Filestores) to allocate the context correctly according to the rules (like, weights, etc).		The standard createcontext call (be it via CLT or RMI or SOAP) is designed to work under all possible cornercases, like concurrent createcontext, createuser, deletecontext, including potential generation and removal of DB schemas and correct counting of schemas on target (DBs, Schemas, Filestores) to allocate the context correctly according to the rules (like weights, etc).

	In order to achieve this, a number of expensive DB queries and extensive locking needs to take place. We optimized this as far as possible while still meeting the "general purupose" requirements outlined above, but it still is costly.		In order to achieve this, a number of expensive DB queries and extensive locking needs to take place. We optimized this as far as possible while still meeting the "general purupose" requirements outlined above, but it still is costly.

Dominik.epple: Created page with "= Context Preprovisioning = The standard createcontext call (be it via CLT or RMI or SOAP) is designed to work under all possible cornercases, like concurrent createcontext,..."

2015-10-01T08:50:59Z

Created page with "= Context Preprovisioning = The standard createcontext call (be it via CLT or RMI or SOAP) is designed to work under all possible cornercases, like concurrent createcontext,..."

New page

= Context Preprovisioning =

The standard createcontext call (be it via CLT or RMI or SOAP) is designed to work under all possible cornercases, like concurrent createcontext, createuser, deletecontext, including potential generation and removal of DB schemas and correct counting of schemas on target (DBs, Schemas, Filestores) to allocate the context correctly according to the rules (like, weights, etc).

In order to achieve this, a number of expensive DB queries and extensive locking needs to take place. We optimized this as far as possible while still meeting the "general purupose" requirements outlined above, but it still is costly.

For other usecases a lot of these queries and locks is not necessary. In particular if it is possible to allocate a "pre-provisioning" phase in an implementation project where only createcontext takes place, and all schemas can be pre-generated, a lot of these allocation queries and locks can be skipped.

This article is to describe the prerequisites of this "fast mode preprovisioning", and how to execute it.

== Prerequisites ==

* All required DB schemas are pre-generated (see below)
* Only createcontext, no other provisioning calls, no HTTP API calls
* Only one provisioning host (which usually does not become the bottleneck; the bottleneck is on the DB usually)

== Execution ==

=== Schema pre-generation ===

To pre-generate schemas, you can temporarily set the setting CONTEXTS_PER_SCHEMA in /opt/open-xchange/etc/plugin/hosting.properties to the value 1. Then, every subsequent createcontext will trigger the generation of a schema. So we recommend to generate the schemas with placeholder contexts which will not actually used for anything, just to generate the schemas, and by not deleting them we make sure the schemas are not teared down also.

So with this setting CONTEXTS_PER_SCHEMA=1, you create as much contexts as you require for your final number of contexts to be provisioned. Then you change that setting back to its original value (as per system high level design, usually something between 1000 and 7000).

=== Context Creation in Fast Mode ===

There are a number of options you need to set to run in "fast mode".

* In order to save OX from counting contexts per filestore, you should use the --destination-store-id <id> argument to specifiy a target filestore.
* The same reasoning applies to the target database, --destination-database-id <destination-database-id>
* Then to use the "fast mode" (skipping locks and some checks) you can use one of the two following. For a discussion see below.
** --schema-name <schema-name>
** --schema-strategy=in-memory

Then, the most important thing is: context creation parallelizes reasonably well at least until the number of DB hosts you got in your setup, perhaps even beyond. So you want to implement some parallel scheme of creating contexts, but with lesser parallelity than the number of schemas you pregenerated. The optimal number of parallel createcontext "streams" depends on your infrastructure and is to be determined heuristically. On single node development machines values up to 10 concurrent streams are reasonable. On larger test platforms (and also on a production platform) we found that up to 100 streams make sense and increase the throughput. But, if increasing the number of streams further does not increase throughput, but only latency, you should reduce the number of streams again.

To implement parallel streams of createcontext, you can do one of the following:

* The most simple (and most simple to understand) approach is to create one huge .csv file which is designed to be fed into the CLT createcontext --csv. Then you split that file into a number of chunks which matches the number of parallel streams you want to run. Then you invoke (from a shell script, for example), just the corresponding number of createcontext --csv tools in parallel.
* More sophisticated is to create a program which creates the required data (or reads them from some input) and uses a multithreaded worker model to create the required number of streams
* You can use our [[Gatling_Performance_Tests|gatling tools]] to load-test your system. You will probably need help setting up this stuff, so contact your OX consultant for assistance.

==== Schema-Name vs Schema-Strategy ====

These are two different implementations with the same goal to run in "fast mode", skipping locks and some checks.

The approach of the --schema-name method is to require the caller to do book-keeping of how many contexts are allocated where. Usually some simple round-robin approach is sufficient here. But be aware that the schema names you supply must match the DB ID you supply. So usually you have a tool which creates the createcontext requests, and this tool needs to be coded such that the schema names it gives are valid for the given DB ID.

The approach of the --schema-strategy=in-memory is to not require any logic from the caller, but rather do the required book-keeping in memory of OX.

Both approaches seem to work in our benchmarks with similar achieved throughput. None of them have undergone extensive real-world usage. Theoretically the in-memory approach is easier to integrate into existing workflows, but the schema-name approach is much simpler inside of OX ant potentially more robust.