.. _plans:

Factory Plans
=============

Simple local plan
~~~~~~~~~~~~~~~~~

The plan below is the simplest factory defines the two bots from pypi source and a single stage called ``test``
and a local connection.


.. literalinclude:: _static/plans/simple-local-plan.yml
    :language: yaml
    :emphasize-lines: 19, 22
    :linenos:

Building this factory with the command line below:

.. code-block:: commandline

    qalx factory-build --plan simple-local-plan.yml --stage test

Will start a ``sim_bot`` in two processes and a ``batch_bot`` in one process. They will read from the specified
queue names in the highlighted lines.

Local multiple stage plan
~~~~~~~~~~~~~~~~~~~~~~~~~

The plan below adds an additional stage to the plan above.


.. literalinclude:: _static/plans/local-multiple-stage-plan.yml
    :language: yaml
    :emphasize-lines: 30,31, 33
    :linenos:

Building this factory with the command line below:

.. code-block:: commandline

    qalx factory-build --plan local-multiple-stage-plan.yml --stage production

Will now start a ``sim_bot`` in eight processes. These will read from a different queue as well.


Remote plan
~~~~~~~~~~~

The plan below shows a stage which will deploy the bots to a server on AWS.

.. literalinclude:: _static/plans/remote-aws-plan.yml
    :language: yaml
    :emphasize-lines: 18-30
    :linenos:

The key parameters are in the highlighted lines:

+ ``ImageId``: this is image needs to be a ``pyqalx`` base image
  (contact us if you don't have access to these) with OrcaFlex installed and with access to a licence.
+ ``InstanceType``: this is the server that the bots will be deployed to.
+ ``KeyName``: if you want to be able to remote desktop to the servers you need to specify an existing key pair name.
+ ``NetworkInterface``: the server will need to be placed in a subnet with access to the internet and
  the licence server. It will also need a security group that allows this and RDP access if required.

.. code-block:: commandline

    qalx factory-build --plan remote-aws-plan.yml --stage test

Multiple clusters
~~~~~~~~~~~~~~~~~

This plan is a more complete example, it allows you to run bots locally during development, then deploy them to a
small cloud server for testing and then offers production clusters in 3 different sizes.

.. literalinclude:: _static/plans/full-cluster-plan.yml
    :language: yaml
    :emphasize-lines: 32, 47
    :linenos:

This plan leverages `anchors and aliases in YAML <https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/>`__
to simplify the plan. The anchors are defined on the highlighted lines and then re-used with aliases lower down the plan.

Calling the command:

.. code-block:: commandline

    qalx factory-build --plan full-cluster-plan.yml --stage cluster_large

Will start 5 servers in AWS with 128 cores each.

Terminating servers
~~~~~~~~~~~~~~~~~~~

You can _demolish_ a factory using the command:

.. code-block:: commandline

    qalx factory-demolish --name my-factory-name

Where ``my-factory-name`` is the name on line 3 of all the plans above.
This command will gracefully terminate the bots on each server and then terminate the server. Each bot should finish the
job is was doing before the server terminates. This is great when you know that your batches are all finished but if
you are running batches overnight you might have some pricey servers sitting idle until someone checks that they are
finished.

The plan below shows how you can leverage
`CloudWatch alarms <https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html>`__
to ensure that each server will terminate (and stop costing money) once it has bene idle for a certain period.

.. literalinclude:: _static/plans/remote-aws-alarm-plan.yml
    :language: yaml
    :emphasize-lines: 18, 41
    :linenos:

In this plan as soon as the server has a CPU utilisation below 2% for ten minutes the server will be terminated.

.. attention::

    The CPU levels that define an "idle" state will vary based on the server size and the jobs the bots are doing.
    You should be careful not to define an alarm that will terminate a server that is finishing off one
    final long-running job. These alarms do not wait for jobs to finish before terminating the server.


Workflows
~~~~~~~~~

Often when a batch completes you will want to process the information in the batch to create plots or populate a report.

The plan below shows how to use the `workflows <https://docs.qalx.net/factories#keys-and-validation-requirements>`__
a feature of factories to send jobs from ``batch_bot`` on to an ``plot_bot``.

.. literalinclude:: _static/plans/simple-workflow-plan.yml
    :language: yaml
    :emphasize-lines: 20, 34-37
    :linenos:

More details on creating custom bots can be found in :ref:`custom_bots`.