Installing the Infoset Generator

Using the Infoset Generator

The Infoset Generator can be used both through a REST API, or a command line interface. The Docker container exposes port 8080 on which HTTP requests are accepted. This port can then be bound to a port on the Docker host to accept requests from other machines.

To use this software you must first obtain a license from 28msec.

Installing the Infoset Generator

Create a folder, referred as <INFOSETGENERATOR_FOLDER> in what follows, on the Docker host to keep its configuration and license files.

mkdir -p <INFOSETGENERATOR_FOLDER>

For instance:

mkdir -p /28msec/infoset-generator

Store the license file in a file named XBRLlicense.lic (case sensitive) in the <INFOSETGENERATOR_FOLDER> folder, for instance /28msec/infoset-generator/XBRLlicense.lic. The Infoset Generator needs to be restarted after changing the license file, as documented in the "Starting/Stopping the Infoset Generator" section.

Obtain the latest Infoset Generator docker container (this requires credentials provided by 28msec):

docker login
docker pull 28msec/infoset-generator:latest

Initialize and run the Infoset Generator Docker instance:

docker run -d -p <INFOSETGENERATOR_PORT>:8080 -v <INFOSETGENERATOR_FOLDER>:/etc/infoset-generator --name infoset-generator 28msec/infoset-generator:latest server <MAX_CONCURRENCY_LEVEL> <MEMORY_LIMIT_MB>

where

is the host port where you want Infoset Generator requests to be accepted,
is the number of requests which can be executed simultaneously,
is the global maximum amount of memory used in MB.

For instance:

docker run -d -p 82:8080 -v /28msec/infoset-generator:/etc/infoset-generator --name infoset-generator 28msec/infoset-generator:latest server 2 8192

This command will initialize and start the Infoset Generator instance.

Optionally, you can install a taxonomy cache to speed up the processing of SEC and FSA filings, following the instructions in the "Installing the default taxonomy cache" section. For other taxonomies you can follow the instructions in the taxonomy cache page.

Check the health of the Docker instance, following the instructions in the "Checking the Docker instance health" section.

Installing the default taxonomy cache

To install the default taxonomy cache you need to download and extract it to the chosen .

cd <INFOSETGENERATOR_FOLDER>
wget https://28federico.s3.amazonaws.com/taxonomy-cache/taxonomy-cache.tar.gz
tar -xvzf taxonomy-cache.tar.gz
rm taxonomy-cache.tar.gz

For instance:

cd /28msec/infoset-generator
wget https://28federico.s3.amazonaws.com/taxonomy-cache/taxonomy-cache.tar.gz
tar -xvzf taxonomy-cache.tar.gz
rm taxonomy-cache.tar.gz

Check the Docker instance health

A quick way to check if the Docker instance started correctly is to execute the following command on the Docker host:

wget https://28msec-test-data.s3.amazonaws.com/xbrl/cs-b-20150630.zip -O /tmp/filing.zip && \
curl --write-out "Response code: %{http_code}\nTotal time: %{time_total} s\n" --silent \
-X POST 'http://localhost:<INFOSETGENERATOR_PORT>/v1/filings?profile=SEC' \
--data-binary "@/tmp/filing.zip" -H "Content-Type: application/xbrlx" -o /dev/null

where is the host port where Infoset Generator requests are accepted.

For instance:

wget https://28msec-test-data.s3.amazonaws.com/xbrl/cs-b-20150630.zip -O /tmp/filing.zip && \
curl --write-out "Response code: %{http_code}\nTotal time: %{time_total} s\n" --silent \
-X POST 'http://localhost:82/v1/filings?profile=SEC' \
--data-binary "@/tmp/filing.zip" -H "Content-Type: application/xbrlx" -o /dev/null

On an healthy machine the following output would be produced:

Response code: 200
Total time: 5.363 s

An unhealthy machine will produce a response code of 500. In this case to inspect the actual error re-run the same command without -o /dev/null at the end.

Startup Logs

To check the startup logs on the host machine you can run docker logs infoset-generator

On an healthy instance the last lines look as follows:

Maximum number of connections 10
Maximum heap size 8192
29-Oct-2015 11:03:14.677 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.0.28
29-Oct-2015 11:03:14.679 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Oct 7 2015 18:25:21 UTC
29-Oct-2015 11:03:14.679 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:         8.0.28.0
29-Oct-2015 11:03:14.679 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
29-Oct-2015 11:03:14.679 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            3.19.0-30-generic
29-Oct-2015 11:03:14.679 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
29-Oct-2015 11:03:14.680 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-7-oracle/jre
29-Oct-2015 11:03:14.680 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.7.0_80-b15
29-Oct-2015 11:03:14.680 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
29-Oct-2015 11:03:14.680 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /opt/tomcat8
29-Oct-2015 11:03:14.680 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /opt/tomcat8
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/tomcat8/conf/logging.properties
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.egd=file:/dev/./urandom
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx8192M
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
29-Oct-2015 11:03:14.681 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line arguwebapps/manager
29-Oct-2015 11:03:18.209 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /opt/tomcat8/webapps/manager has finished in 38 ms
29-Oct-2015 11:03:18.217 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
29-Oct-2015 11:03:18.226 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
29-Oct-2015 11:03:18.227 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3330 ms

The most important ones are the first two (which show the configuration options) and the last one, which confirms that the instance started successfully.

Starting the Infoset Generator automatically at boot time

Ensure that all needed filesystem are already mounted at service start time. On systemd systems you can use systemctl to check the names of mount services.

Systemd

Create infoset-generator service:

sudo tee /etc/systemd/system/infoset-generator.service <<-EOF
[Unit]
Description=28msec Infoset Generator
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=30
TimeoutStartSec=60
TimeoutStopSec=300
ExecStart=/usr/bin/docker start -a infoset-generator
ExecStop=/usr/bin/docker stop -t 300 infoset-generator

[Install]
WantedBy=multi-user.target
EOF

Enable and start service:

sudo chkconfig infoset-generator on
sudo systemctl start infoset-generator

Upstart

sudo tee /etc/init/infoset-generator.conf <<-EOF
description "28msec Infoset Generator"
start on filesystem and started docker
stop on runlevel [!2345]

normal exit 0
respawn
respawn limit unlimited
kill timeout 60

script
  /usr/bin/docker start -a infoset-generator
end script
pre-stop script
  /usr/bin/docker stop -t 60 infoset-generator
end script
EOF

Start service:

sudo service infoset-generator start

Starting/Stopping the Infoset Generator

When the Infoset Generator is initialized using the docker run command it is started automatically. To start/stop the Infoset Generator you can use the following commands:

If you did not create a service:

docker stop infoset-generator
docker start infoset-generator

If you created a systemd service:

sudo systemctl stop infoset-generator
sudo systemctl start infoset-generator

If you created a upstart service:

sudo service infoset-generator stop
sudo service infoset-generator start

Updating the Infoset Generator

Stop the Infoset Generator container if it is running, as described in the "Starting/Stopping the Infoset Generator" section.
Remove the installed Infoset Generator container. The user data will not be removed.
```
docker rm infoset-generator
```
Follow the instructions in the "Installing the Infoset Generator" section.