1   Introduction¶

The Vera Rubin Observatory control system is highly distributed, containing multiple components that must operate in consonant. In order to achieve reliable operation it is fundamental to guarantee that observers must be able to, at any point, identify what set of configurations a component is using, what sets are available to be loaded, and how to load the configuration they choose. During commissioning and engineering runs, users will also be interested in building, validating and verifying new sets of configurations.

Although the configuration data structures (e.g. schemas) vary considerably between components, the process to update them is standardized. Being CSC specific, the schemas exist within the CSC code repository as a config_schema.py file, as is seen here for the ATDome CSC. In some cases, the configuration may be a simple set of host name and port that the component connects to. In other cases, the configuration may require the calculation of complex lookup tables (e.g. M1M3 and M2) or model coefficients (e.g. pointing component). These may require on-sky time for acquiring the data, and complex software to analyze and derive the configuration products. All of this data needs to be associated with this configuration in a way that is easily accessible and traceable.

In this tech-note we discuss the basic organization of configurations and the operational procedures involved in handling CSC configuration. The document begins with the most simple case of bringing a CSC up with a specific configuration, and eventually details how to update and test newly defined configurations.

2   Configuration Interface Summary¶

The standard interface for CSCs is defined in LSE-209 and is the official reference/requirements document. This section provides a very brief overview of the key topics related to configuration that are described in further detail in the sections below.

Configuration-related interactions with CSCs happens via two main events (configurationsAvailable and configurationApplied) and a single parameter as part of the start command (configurationOverride). When in the STANDBY state, the configurationsAvailable event is published which includes a list of non-default (custom) configurations as well as information required to track available configurations (both default and custom) to a specific git commit. Under normal operations, site-specific default configurations are automatically loaded. However, using the configurationOverride parameter (explained in detail in Selecting an Override Configuration_ section), it is possible to specify one of the non-default configurations that is listed in the configurationsAvailable event. Lastly, the configurationApplied event is published when the CSC transitions from the STANDBY state to the DISABLED state (which is on the way to the ENABLED state). The event contains the information regarding which configuration(s) have been loaded and are in active use by the CSC.

3   Configuration Files and Repositories¶

It is important to have the ability to modify configurations without re-deploying components, therefore, configuration files are stored in their own repositories and kept separated from the code. The configuration repository associated with each CSC is found in the configuration column of the Master CSC Table. Each of these configuration repositories are organized as follows:

• A single configuration repository may host configurations for multiple CSCs.

• CSC configurations are stored in folders with the CSC name.

• Each CSC folder contains sub-folders corresponding to the versions of the CSC configuration (e.g. v1, v2).

• A folder for a given version contains up to three different types of configuration files, all of which are yaml files. The three types of configuration files are detailed below and are listed in the order of which they are read:

  1. Initial Configuration: _init.yaml.

     • This required file contains all values that are expected to be common to all sites and/or be relatively static in operations. This file may contain a complete set of parameters, but is only required to do so if no site-specific configuration file exists.

  2. Site-specific Configuration: _summit.yaml, _ncsa.yaml, _base.yaml etc.

     • This optional file contains contain site specific configuration parameters such as IP addresses and ports. Many CSCs have site specific files. SalObj determines which site-specific file should be loaded automatically by parsing the LSST_DDS_PARTITION_PREFIX environment variable Between this file and the _init.yaml file, the configuration must be fully defined

  3. Configuration overrides: filename.yaml

     • These optional files, referred to as configuration overrides, are only to be used when the values declared in the previous files require changes. These files are loaded manually by the users as is demonstrated in the Selecting an Override Configuration section.

• If a value is specified in more than one of these files, the most recently seen value is used. This means that values in the site-specific (_<site>.yaml) file override values in the initial file (_init.yaml). Also, values in the override file (filename.yaml) override values populated in the _init.yaml and _<site>.yaml files.

4   Starting CSCs with Existing Configurations¶

Users will interact with configurations in multiple ways. In many cases, a user/operator will only need to change the configuration that is currently loaded and are not concerned with the contents of the configuration itself. In other cases, the user/operator will need to make a change to file, then immediately reload it. This section illustrates example use-cases for these types of scenarios.

from lsst.ts.observatory.control import ATCS

atcs = ATCS()

await atcs.start_task

await atcs.enable()

from lsst.ts import salobj

domain = salobj.Domain()

atdome = salobj.Remote(domain, "ATDome")

await atdome.start_task()

# CSC needs to be in STANDBY state for this to work
await salobj.set_summary_state(atdome, salobj.State.ENABLED)

from lsst.ts import salobj

domain = salobj.Domain()

atdome = salobj.Remote(domain, "ATDome")

await atdome.start_task()

config_available = await atdome.evt_configurationsAvailable.aget(timeout=5.)

# This will print the available filenames.
print(config_available.configurations)

# This will print the git hash of the loaded configuration repository
print(config_available.version)

from lsst.ts.observatory.control import ATCS

atcs = ATCS()

await atcs.start_task

config_available = await atcs.rem.atdome.evt_configurationsAvailable.aget(timeout=5.)

# This will print the available filenames.
print(config_available.configurations)

# This will print the git hash of the loaded configuration repository
print(config_available.version)

from lsst.ts.observatory.control import ATCS

atcs = ATCS()

await atcs.start_task

# ATAOS must be in STANDBY state for this to work. All other CSCs will
# use their default configurations
await atcs.enable(override={'ATAOS': 'summit_constant_hex.yaml'})

from lsst.ts import salobj

d = salobj.Domain()

atdome = salobj.Remote(d, "ATDome")

await atdome.start_task()

await salobj.set_summary_state(
atdome, salobj.State.ENABLED, override="simple_algorithm.yaml"
)

5   Modifying or Creating a New Configuration¶

The process to derive new configuration parameters varies considerably from component to component. In some cases, the configuration is simple enough that a change may involve simply replacing an IP or hostname value, a routine filter swap on an instrument or updating the limits to an axis range due to some evolving condition. On the other hand, deriving new parameters may involve generating complex LUTs that may require on sky observations and detailed data analysis.

Following is a detail of each step of the process to generate a new configuration and update it for CSCs written in salobj. For other components, see the exception section below.

1. Create a Jira ticket to track the work being done (e.g. DM-12345). If details or discussions are needed they can done using the Jira ticket itself. Then clone the configuration repository and create a new branch corresponding to the Jira ticket number.

   git clone git@github.com:lsst-ts/ts_config_attcs.git
   git checkout -b tickets/DM-12345
   

2. Execute the work needed to derive the new configuration parameter(s).

   As mentioned above, in some cases, the process may be straightforward, consisting simply of replacing the values of a set of parameters with given values (e.g., swapping filters). In these cases, this step will be simply verifying any required work was performed and continuing to the next step. Jira should be used to track those activities.

   The Jira ticket should also be used to track the work done on those cases where a more involved analysis is required, e.g., in-dome and/or on-sky data acquisition, EFD queries, data processing etc. Any ancillary software or data product required during this process should be properly managed using git. When working with Telescope and Site components, any software required during this process should be stored in a git repository in T&S GitHub organization, and should follow the standard T&S development workflow guidelines. This includes, but is not limited to, EFD queries, Jupyter notebooks, other data analysis routines (regardless of the programming language) and so on. The preferred location for storing Jupyter notebooks is the ts_notebooks repository. If the procedure to generate the new configuration requires detailed explanation, a tech-note in tstn repository can be created and linked to the ticket.

   Any intermediate data product(s) generated in the process should also be stored in the git Large File Storage or, if size permits, with the software repository itself.

3. Edit/Add/Replace the configuration file(s) in the CSC’s configuration directory.

   • If editing the _init.yaml or a _<site>.yaml file, the filename must remain unchanged.
   • If editing or adding a configuration override file, the name of the file should reflect the purpose of change. Including the date as part of the name is also recommended. Old configuration files can be kept in the repo if they still represent valid configurations. Otherwise, they should be removed. Note, though, that they will still remain available on previous commits in the git repo, enabling historical comparison.

4. Fill out the required metadata at the top of the file detailing where any auxiliary data may be stored, the Jira ticket number used to create the file, and the reason for creating the configuration, such as in this example.

5. If you have an environment to do so, such as the standard T&S development container, run the unit tests in the package locally.

6. Add, commit and push the changes, with a commit message.

   git commit -am "Add new LUTs for ATAOS (file 20200512-configuration.yaml) based on data taken on 20200512. Check DM-12345 for more information."
   git push
   

7. Verify the continuous integration tests pass. If they don’t, fix the issue and repeat the previous step.

8. Test the new configuration on the CSC. If this requires in-dome or on-sky testing, then create an annotated alpha release tag. Then make sure the test is properly documented in a technote and/or Jira ticket. To make the configuration available on a running CSC check On-the-Fly Configuration Changes.

9. Create pull request(s) (PRs) to have the files reviewed

   You must create PRs for all repositories that were modified during the process, including, but not limited to, the configuration repository, ancillary software and documentation.

   Once the PRs are reviewed and approved, the files can be merged and subsequently tagged. The new configuration then becomes official and will be deployed as part of the standard deployment process.

6   On-the-Fly Configuration Changes¶

During the process of creating a new configuration (see also Modifying or Creating a New Configuration) or during a commissioning/engineering run, it may be necessary to make a new configuration available to a running CSC. Normally, new configurations are only made available when rebuilding/re-deploying the component and associated configuration repository, but this is not always feasible. Therefore, in cases where on-the-fly configuration changes are required, the following procedure should be followed to make a new configuration available to a running CSC:

1. If the configuration is not already created and pushed to GitHub, follow steps 1 to 8 in Modifying or Creating a New Configuration.

2. Create an annotated tag alpha tag following semantic versioning. The tag must be created to ensure the heritage is not lost in a forced commit to the branch

   git tag -a v1.4.0.alpha.1 -m "Updated focus values based on on-sky tests"
   

3. Make sure the CSC in in STANDBY state, which can be accomplished using the following command.

   await salobj.set_summary_state(ataos, salobj.State.STANDBY)
   

4. Login to the where the CSC is running. The procedure will vary depending on how the CSC is deployed. Most Telescope and Site components are deployed on containers using Kubernetes (k8s). For CSCs that are not running on a container, you should be able to login to the host machine with ssh and continue with the procedure (go to step 3). A provisional list of IPs can be found in confluence. For details about the deployment system see the deployment documentation.

   The procedure to access containerized components is as follows:

   1. Log in to the rancher service at https://rancher.ls.lsst.org. You will need special authorization to acquire an account on that service.

      Warning

      This service is responsible for managing the deployment of the entire system. Make sure you follow the procedure exactly. If you are in doubt about an operation make sure you verify it with knowledgeable personnel.

   2. Once logged in, you will be presented with the list of available k8s clusters.

      

      Figure 1 List of Kubernetes clusters. At the time of this writing, the only cluster available was kueyen, the commissioning cluster at the base facility in Chile.

      Click on the name of the cluster where the CSC you want to modify is running. If it is a summit operation, the name of the cluster will be andes. After selecting the cluster, you will be redirected to the cluster dashboard.

      

      Figure 2 Cluster dashboard.

   3. On the top right corner of the cluster dashboard, there is a button with Launch kubectl. This will open an interactive session on you browser that will allow you to interact with the k8s cluster you selected. If you are knowledgeable about k8s you can also download the Kubeconfig file and login to the cluster from your own computer.

      Warning

      Do not download the Kubeconfig file unless you really know what you are doing. This file contains access and credential information that would allow users direct access to the k8s cluster.

   4. Once you select Launch kubectl you will be redirected to a Shell connected directly to the selected k8s cluster.

      

      Figure 3 Kubectl shell.

   5. Use the following command to discover the container running the CSC :

      kubectl get pods -n cscs
      

      This will list all the CSCs “pods” which are, basically, the running containers. The name of the CSC will be part of the pod name and should be easy to identify.

   6. Connect to the running pod:

      kubectl exec -it -n cscs <pod-name> -- /bin/bash
      

      Make sure to replace <pod-name> with the name of the pod for that CSC.

5. Once inside the CSC host, go to the location where the configuration is installed. This information can be found in the CSC documentation or in the deployment documentation. You should be able to use regular linux command line commands (e.g. ls and cd).

6. Once in the cloned configuration package, update the git repository and checkout the tag with the new configuration:

   git fetch --all
   git checkout tags/v1.4.0.alpha.1
   

   You should see the new tag be pulled and git will tell you that you’ve changed tags/branches.

7. Now re-enable the component to load the new configuration.

   If the _init.yaml or _<site>.yaml file was modified then use the following:

   await salobj.set_summary_state(ataos, salobj.State.ENABLED)
   

   If an override configuration was modified/added, then you must specify it using the override keyword

   await salobj.set_summary_state(ataos, salobj.State.ENABLED, override='summit_constant_hex.yaml')
   

The version attribute in the configurationsAvailable event would reflect that change with something like:

version: heads/tags/v1.4.0.alpha.1-g79e2257

Note that it would be possible to track the configuration in the future by using the commit hash (g79e2257).

version: heads/tickets/DM-12345-0-g79e2257-dirty

7   Finding and Using a Previously Used Configuration¶

In the future, one may want to verify which configuration was being used for a given observation and possibly load the exact same configuration. Because we often use generic filenames (e.g. simple_algorithm.yaml), and file contents can change with time, creating a robust version controlled system must go beyond changing filenames. For this reason, additional metadata is associated with each configuration, notably the url and version parameters in both the configurationsAvailable and configurationApplied events. These parameters are key to ensuring that each configuration is unique, and is traceable to their filename and contents.

The url parameter contains a URL indicating how the CSC connects to its configuration repository.

The version parameter is a bit more complicated. For all CSCs (except possibly the cameras), the version parameter is a branch description[1] which is automatically generated and populated by the CSCs. It can be obtained by running the following git command on the command line. The following example and output was derived using a real world example, however, the files are no longer in place and therefore will not yield the same results.

git describe --all --long --always --dirty --broken

When running the command in a configuration repository (e.g. ts_config_latiss) the output is, heads/develop-0-gc89ef1a. The repository branch (or tag) name forms the first part of the branch description. This first part (heads/develop-0) contains individual identifiers that may change. It may take any form necessary to convey the appropriate information. The last 7 characters (c89ef1a) is the hash of the commit of repository, so all configuration files in that repo correspond to the same hash. Users can find this commit by navigating to the repository on github, searching for the commit hash, then clicking on the “commits” section of the search results, as shown in the screenshot below.

Figure 4 Using the version output in the configurationApplied event, it is possible to trace configuration that was loaded back to a specific commit to the configuration repository.

Once we have identified the hash of the commit file we want to reload, we can do that without having to make any changes to the currently deployed ts_config package. If we simply want to use the default site-specific configuration for a given CSC, we can specify the commit hash with a preceding colon (:) as follows:

from lsst.ts.observatory.control import ATCS

atcs = ATCS()

await atcs.start_task

# ATAOS must be in STANDBY state for this to work. All other CSCs will
# use their default configurations
await atcs.enable(configurationOverride={'ATAOS': ':c89ef1a'})

If we also want to specify an override file then we insert the filename before the colon (:) as shown below:

await atcs.enable(configurationOverride={'ATAOS': 'simple_algorithm.yaml:c89ef1a'})

8   CSC Developer Information¶

This section contains information that is primarily of interest to CSC developers. However, if issues are being encountered when creating new configurations the information may be pertinent.

9   Appendix I: Creating Configurations for non-salObj CSCs¶

This appendix explains how to produce configuration files for CSCs that do not fully follow the standard above.

10   Appendix II: System Requirements¶

This document is derived from requirements spread accross multiple resources. Many come from LSE-209. The following are requirements from other system level requirement documents that are also applicable.