Installing the CellStore

Configuring the CellStore

The first step is to create a folder, named <CELLSTORE_FOLDER> in what follows, that will contain the CellStore configuration. The configuration file must be called docker-config.json.

For example:

mkdir /28msec/cellstore
touch /28msec/cellstore/docker-config.json

In docker-config.json you need to specify the configuration in which you would like the CellStore to be installed. Below is the expected format of such a file.

{
    "28": {
        "email": "<Platform account email>",
        "password": "<Platform account password>",
        "platformEndpoint": "<Platform API endpoint for docker>",
        "projectEndpoint": "<CellStore API endpoint for docker>"
    },
    "cellstore": {
        "endpoint": "<CellStore API endpoint for clients>",
        "projectName": "<CellStore project name>",
        "datasources": [<CellStore datasources>],

        "profile": "<CellStore default profile>",

        "adminUser": "<CellStore admin email>",
        "adminPassword": "<CellStore admin password>",
        "adminToken": "<CellStore admin API token>",

        "testUser": "<CellStore read-only user email>",
        "testPassword": "<CellStore read-only user password>",
        "testToken": "<CellStore read-only user API token>",
        "infosetGenerator": {
          "URL": "<Infoset generator endpoint. (optional)>",
          "timeout": "<Infoset generator timeout in seconds. (optional)>"
        }
    }
}
  • 28.email: Email associated with an account on the 28msec platform. Examples of values: admin (default) or mymail@gmail.com.
  • 28.password: Password of the 28msec platform account.
  • 28.platformEndpoint: API endpoint at which the 28mec platform is reachable by this docker container. This value is in the format http://<PLATFORM_ENDPOINT>/portal/api. For instance: http://28.28.28.28:81/portal/api, or http://myserver.com/portal/api.

  • 28.projectEndpoint: API endpoint at which the CellStore project is reachable by this docker container. This value is in the format http://<PLATFORM_ENDPOINT>/<CELLSTORE_PROJECTNAME>/v1, where is the value defined for cellstore.projectName. Example of values: http://28.28.28.28:81/mycellstore/v1, or http://myserver.com/mycellstore/v1.

  • cellstore.projectName: Project name under which you would like to deploy the CellStore. Example of value: mycellstore. This name is used to identify the API endpoint at which the CellStore is reachable.

  • cellstore.endpoint: API endpoint at which the CellStore project can be reached by the CellStore users. This value is in the format http://<PLATFORM_ENDPOINT>/<CELLSTORE_PROJECTNAME>/v1, where is the value defined for cellstore.projectName. Example of values: http://28.28.28.28:81/mycellstore/v1, or http://myserver.com/mycellstore/v1.
  • cellstore.datasources: Databases used by the CellStore. Below are two examples of datasources configuration. In the first example, the CellStore is configured with a MongoDB and an ElasticSearch databases. The values for the mapping object are provided by 28msec. In the second example, a single MarkLogic database is used.
{
    "category": "MongoDB",
    "name": "xbrl",
    "types": ["query"],
    "credentials": {
        "conn-string": "mongodb.mydomain.com",
        "db": "dbname",
        "user": "user",
        "pass": "password"
    }
}, {
    "category": "ElasticSearch",
    "name": "xbrl",
    "types": [ "search" ],
    "credentials": {
        "urls" : [ "54.65.202.1:9200", "54.65.202.2:9200" ],
        "protocol": "https",
        "db": "dbname",
        "user": "user",
        "pass": "password",
        "verify-peer": false,
        "verify-host": false,
    },
    "mapping": {          
        "concepts": "copy",
        "entities": "copy"
    }
}

{
    "category": "MarkLogic",
    "name": "xbrl",
    "types": ["query", "search"],
    "credentials": {
        "username": "user",
        "password": "password",
        "hostname": "52.69.96.1",
        "port": 8003
    }
}
  • cellstore.profile: CellStore default value for the profile parameter in the REST API. Value provided by 28msec. Example of value: japan, dpm, or sec.
  • cellstore.adminUser: Email of the CellStore admin user.
  • cellstore.adminPassword: Password of the CellStore admin user.

  • cellstore.adminToken: CellStore admin API token.

  • cellstore.testUser: Email of the CellStore read-only user.

  • cellstore.testPassword: Password of the CellStore read-only user.

  • cellstore.testToken: CellStore read-only user API token.

  • cellstore.infosetGenerator.URL: API endpoint of the Infoset Generator (optional). The Infoset Generator is the component used to insert an XBRL filing into the CellStore.

  • cellstore.infosetGenerator.timeout: Timeout in seconds of the Infoset Generator (optional).

Once the configuration file is ready, you can pull the latest version of the CellStore: 

docker pull 28msec/cellstore:v26.8.0

Initializing MongoDB databases

To use a MongoDB database a set of indexes must be created.

Indexes can be created in background or in foreground. When indexes are created in foreground the performance of the entire database machine is very severely degraded and concurrent operations can be blocked, but indexes are created faster and can be more compact. If the database is empty or the indexes already exists the index creation will take a few seconds. Otherwise the index creation might take a very long time. For this reason, by default, indexes are created in background.

The index definitions can be different in new releases and among profiles, therefore the following process should be repeated when installing a new version of the CellStore or changing the CellStore profile.

Execute the following command in order to create the indexes in background:

docker run --rm -it -v <CELLSTORE_FOLDER>:/etc/cellstore 28msec/cellstore:v26.8.0 create-mongodb-indexes

For instance:

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 create-mongodb-indexes

Alternatively, execute the following command in order to create the indexes in foreground:

docker run --rm -it -v <CELLSTORE_FOLDER>:/etc/cellstore 28msec/cellstore:v26.8.0 create-mongodb-indexes --foreground

For instance:

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 create-mongodb-indexes --foreground

Initializing ElasticSearch databases

To use an ElasticSearch database a set of mappings must be created and the data must be synchronized.

The index definitions can be different in new releases and among profiles, therefore the following process might need to be repeated when installing a new version of the CellStore or changing the CellStore ElasticSearch mapping profiles. Since the process is time consuming, it should be run only when instructed by 28msec.

Execute the following commands in order to recreate the mapping and synchronize the data:

docker run --rm -it -v <CELLSTORE_FOLDER>:/etc/cellstore 28msec/cellstore:v26.8.0 reset-elasticsearch
docker run --rm -it -v <CELLSTORE_FOLDER>:/etc/cellstore 28msec/cellstore:v26.8.0 resync-elasticsearch

For instance:

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 reset-elasticsearch
docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 resync-elasticsearch

Install the CellStore

Make sure that the 28msec Platform referenced in the configuration file is running. Execute the following command in order to install the CellStore:

docker run --rm -it -v <CELLSTORE_FOLDER>:/etc/cellstore 28msec/cellstore:v26.8.0 install

For instance:

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 install

It will take a few minutes for the CellStore to be installed. Any error encountered during the installation will be printed out on the console. If the installation has been successfull you will se the following output:

#
# CellStore installation succeeded.
#

Check the CellStore installation health

A quick way to check the Docker instance health is to execute the following command on the Docker host:

curl '<CELLSTORE_ENDPOINT>/_queries/public/api/entities.jq?token=<ADMIN_TOKEN>' --write-out "Response code: %{http_code}\n" -o /dev/null --silent

where CELLSTORE_ENDPOINT is the CellStore project endpoint.

For instance:

curl 'http://localhost:81/cellstore/v1/_queries/public/api/entities.jq?token=admin-token' --write-out "Response code: %{http_code}\n" -o /dev/null --silent

On an healthy machine one of the following two outputs would be produced:

Response code: 200

or

Response code: 204

An unhealthy machine will produce a response code of 500. In this case to inspect the actual error re-run the same command without --write-out "Response code: %{http_code}\n" -o /dev/null --silent at the end.

Running Tests on the CellStore

In addition of checking the health of the CellStore, you can run arbitrary tests against the CellStore installation. To list the available tests: 

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 list-tests

Which produces an output similar to the one below:

[18:17:50] List of available tests:
* tests/rest/dpm/test-suite.json
* tests/rest/edinet/test-suite.json
* tests/rest/risk-data/test-suite.json

You can pick a test from the list and run it with the following command:

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 test <TEST>

For example:

docker run --rm -it -v /28msec/cellstore:/etc/cellstore 28msec/cellstore:v26.8.0 test tests/rest/dpm/test-suite.json

Updating the CellStore configuration

Whenever the CellStore configuration is updated, install the CellStore following the instructions in the "Install the CellStore" section.