Configuration Service

The Configuration Service provides a centralized persistent store for any configuration file used in the TMT Software System. All versions of configuration files are retained, providing a historical record of each configuration file.

Note that in order to use the APIs described here, the Location Service (csw-location-server) and Configuration Service Server needs to be running somewhere in the local network and the necessary configuration, environment variables or system properties should be defined to point to the correct host and port number(s) for the Location Service nodes.

This service will be part of the observatory cluster and exposes Rest endpoints that can be accessed over HTTP. Component developers can use the csw-config-client library in their code. The library wraps the low level communication with Configuration Service Server and exposes simple to use methods to access and manage configuration files.

Dependencies

To use the Configuration Service without using the framework, add this to your build.sbt file:

sbt

libraryDependencies += "com.github.tmtsoftware.csw" %% "csw-config-client" % "5.0.1"

Rules and Checks

• The config file path must not contain !#<>$%&'@^``~+,;= or any whitespace character
• If the input file is > 10MB or has lot of non ASCII characters, then for optimization, server will archive it in annex store.
• Large and binary files can be forced to go to the ‘annex’ store by using a annex=true flag in the create operation.
• API functions accept date-time values in UTC timezone. (e.g. 2017-05-17T08:00:24.246Z)

Model Classes

• ConfigData : Represents the contents of the files being managed. It wraps a stream of ByteString.
• ConfigFileInfo : Represents information about a config file stored in the Config Service.
• ConfigFileRevision : Represents information about a specific version of a config file.
• ConfigId : Represents an identifier associated with a revision of a configuration file, often generated by create or update methods.
• ConfigMetadata : Represents metadata information about the Config Server.
• FileType : Represents the type of storage for a configuration file. Currently two types are supported Normal(small, text files) and Annex(Large, Binary files).

API Flavors

The Configuration Service is used to provide the runtime settings for components. When a component is started, it will use a limited “clientAPI” to obtain the “active” configuration from the Configuration Service, and use those settings for its execution.

To change the active configuration, an administrative tool with access to the full “admin API” must be used. These tools would have the ability to create, delete, and update configurations, as well as retrieve past configurations and their history. Any time a new configuration is to be used by a component, the user must use one of these tools (via CLI, perhaps) to set the active configuration for a component. Since a history of active configurations is maintained by the service, the settings of each component each time it is run can be retrieved, and the system configuration at any moment can be recreated.

Some of the methods provided by the admin API, the methods that change the Config Server content, are protected so that a record can be kept about who is making the modifications. This is done by using a token obtained from the Authentication and Authorization Service. See Admin Protected Routes for more information.

• clientAPI : Must be used in Assembly and HCD components. Available functions are: {exists | getActive}
• adminAPI : Full functionality exposed by Configuration Service Server is available with this API. Expected to be used administrators. Available functions are: {create | update | getById | getLatest | getByTime | delete | list | history | historyActive | setActiveVersion | resetActiveVersion | getActiveVersion | getActiveByTime | getMetadata | exists | getActive}

Accessing clientAPI and adminAPI

The ConfigClientFactory exposes functions to get the clientAPI and adminAPI. Both the functions require the Location Service instance which is used to resolve the ConfigServer.

Scala

copysource// config client API
val clientApi: ConfigClientService = ConfigClientFactory.clientApi(actorSystem, locationService)
// config admin API
val adminApi: ConfigService = ConfigClientFactory.adminApi(actorSystem, locationService, factory)

Java

copysource//config client API
final IConfigClientService clientApi = JConfigClientFactory.clientApi(actorSystem, clientLocationService);
//config admin API
final IConfigService adminApi = JConfigClientFactory.adminApi(actorSystem, clientLocationService, mocks.factory());

exists

This function checks if the file exists at specified path in the repository. It returns Future of a Boolean, whether the file exists or not.

Scala

copysource// construct the path
val filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf")

val doneF = async {
  // create file using admin API
  await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "First commit"))

  // check if file exists with config service
  val exists: Boolean = await(clientApi.exists(filePath))
  // exists returns true/false

Java

copysourcePath filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf");

// create file using admin API
adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "commit config file").get();

Boolean exists = clientApi.exists(filePath).get();

getActive

This function retrieves the currently active file for a given path from config service. It returns a Future of Option of ConfigData.

Scala

copysourceval defaultStrConf: String = "foo { bar { baz : 1234 } }"
val doneF = async {
  // construct the path
  val filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf")

  await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "First commit"))

  val activeFile: Option[ConfigData] = await(clientApi.getActive(filePath))
  // activeFile.get returns content of defaultStrConf

Java

copysourcefinal String defaultStrConf = "foo { bar { baz : 1234 } }";
// construct the path
Path filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf");

adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "First commit").get();

ConfigData activeFile = clientApi.getActive(filePath).get().orElseThrow();
String value = activeFile.toJConfigObject(actorSystem).get().getString("foo.bar.baz");
// value = 1234

create

Takes input ConfigData and creates the configuration in the repository at a specified path

Scala

copysourceasync {
  // construct ConfigData from String containing ASCII text
  val configString: String =
    """
  // Name: ComponentType ConnectionType
  {
    name: lgsTromboneHCD
    type: Hcd
    connectionType: [akka]
  }
  """.stripMargin
  val config1: ConfigData = ConfigData.fromString(configString)

  // construct ConfigData from a local file containing binary data
  val srcFilePath         = ResourceReader.copyToTmp("/smallBinary.bin")
  val config2: ConfigData = ConfigData.fromPath(srcFilePath)

  // construct ConfigData from Array[Byte] by reading a local file
  val stream: InputStream    = getClass.getClassLoader.getResourceAsStream("smallBinary.bin")
  def byteArray: Array[Byte] = LazyList.continually(stream.read).takeWhile(_ != -1).map(_.toByte).toArray
  val config3                = ConfigData.fromBytes(byteArray)

  // store the config, at a specified path as normal text file
  val id1: ConfigId =
    await(adminApi.create(Paths.get("/hcd/trombone/overnight.conf"), config1, annex = false, "review done"))

  // store the config, at a specified path as a binary file in annex store
  val id2: ConfigId =
    await(adminApi.create(Paths.get("/hcd/trombone/firmware.bin"), config2, annex = true, "smoke test done"))

  // store the config, at a specified path as a binary file in annex store
  val id3: ConfigId =
    await(adminApi.create(Paths.get("/hcd/trombone/debug.bin"), config3, annex = true, "new file from vendor"))
  // id1 returns ConfigId(1)
  // id2 returns ConfigId(3)
  // id3 returns ConfigId(5)

Java

copysource//construct ConfigData from String containing ASCII text
String configString = "axisName11111 = tromboneAxis\naxisName22222 = tromboneAxis2\naxisName3 = tromboneAxis3333";
ConfigData config1 = ConfigData.fromString(configString);

//construct ConfigData from a local file containing binary data
URI srcFilePath = getClass().getClassLoader().getResource("smallBinary.bin").toURI();
ConfigData config2 = ConfigData.fromPath(Paths.get(srcFilePath));

ConfigId id1 = adminApi.create(Paths.get("/hcd/trombone/overnight.conf"), config1, false, "review done").get();
ConfigId id2 = adminApi.create(Paths.get("/hcd/trombone/firmware.bin"), config2, true, "smoke test done").get();
//id1 = 1
//id2 = 3

update

Takes input ConfigData and overwrites the configuration specified in the repository

Scala

copysourceval futU = async {
  val destPath = Paths.get("/hcd/trombone/debug.bin")
  val newId = await(
    adminApi
      .update(destPath, ConfigData.fromString(defaultStrConf), comment = "debug statements")
  )
  // newId returns ConfigId(7)

Java

copysourcePath destPath = Paths.get("/hcd/trombone/overnight.conf");
ConfigId newId = adminApi.update(destPath, ConfigData.fromString(defaultStrConf), "added debug statements").get();
//newId = 5

delete

Deletes a file located at specified path in the repository

Scala

copysourceval futD = async {
  val unwantedFilePath = Paths.get("/hcd/trombone/debug.bin")
  await(adminApi.delete(unwantedFilePath, "no longer needed"))
  // validates the file is deleted
  val maybeConfigData = await(adminApi.getLatest(unwantedFilePath))
  // maybeConfigData returns None

Java

copysourcePath unwantedFilePath = Paths.get("/hcd/trombone/overnight.conf");
adminApi.delete(unwantedFilePath, "no longer needed").get();
Optional<ConfigData> expected = adminApi.getLatest(unwantedFilePath).get();
// expected = Optional.empty()

getById

Returns the file at a given path and matching revision Id

Scala

copysourceval doneF = async {
  // create a file using API first
  val filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf")
  val id: ConfigId =
    await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "First commit"))

  // validate
  val actualData = await(adminApi.getById(filePath, id)).get
  // actualData returns defaultStrConf

Java

copysourcePath filePath = Paths.get("/tmt/trombone/assembly/hcd.conf");
ConfigId id = adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "First commit").get();

ConfigData actualData = adminApi.getById(filePath, id).get().orElseThrow();
// actualData.toJStringF(actorSystem).get() = defaultStrConf

getLatest

Returns the latest version of the file stored at the given path.

Scala

copysourceval assertionF: Future[Assertion] = async {
  // create a file
  val filePath = Paths.get("/test.conf")
  await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "initial configuration"))

  // override the contents
  val newContent = "I changed the contents!!!"
  await(adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!"))

  // get the latest file
  val newConfigData = await(adminApi.getLatest(filePath)).get
  // validate
  // newConfigData returns newContent

Java

copysource//create a file
Path filePath = Paths.get("/test.conf");
adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "initial configuration").get();

//override the contents
String newContent = "I changed the contents!!!";
adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!").get();

//get the latest file
ConfigData newConfigData = adminApi.getLatest(filePath).get().orElseThrow();
String filePathContent = newConfigData.toJStringF(actorSystem).get();
//filePathContent = newContent

getByTime

Gets the file at the given path as it existed at a given time-instance. Note:

• If time-instance is before the file was created, the initial version is returned.
• If time-instance is after the last change, the most recent version is returned.

Scala

copysourceval assertionF = async {
  val tInitial = Instant.MIN
  // create a file
  val filePath = Paths.get("/a/b/c/test.conf")
  await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "initial configuration"))

  // override the contents
  val newContent = "I changed the contents!!!"
  await(adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!"))

  val initialData: ConfigData = await(adminApi.getByTime(filePath, tInitial)).get
  // initialData returns defaultStrConf

  val latestData = await(adminApi.getByTime(filePath, Instant.now())).get
  // latestData returns defaultStrConf

Java

copysourceInstant tInitial = Instant.now();

//create a file
Path filePath = Paths.get("/test.conf");
adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "initial configuration").get();

//override the contents
String newContent = "I changed the contents!!!";
adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!").get();

ConfigData initialData = adminApi.getByTime(filePath, tInitial).get().orElseThrow();
//initialData.toJStringF(actorSystem).get() = defaultStrConf
ConfigData latestData = adminApi.getByTime(filePath, Instant.now()).get().orElseThrow();
//latestData.toJStringF(actorSystem).get() = newContent

list

For a given FileType (Annex or Normal) and an optional pattern string, it will list all files whose path matches the given pattern. Some pattern examples are: “/path/hcd/*.*”, “a/b/c/d.*”, “.*.conf”, “.*hcd.*”

Scala

copysource// Here's a list of tuples containing FilePath and FileTyepe(Annex / Normal)
val paths: List[(Path, FileType)] = List[(Path, FileType)](
  (Paths.get("a/c/trombone.conf"), Annex),
  (Paths.get("a/b/c/hcd/hcd.conf"), Normal),
  (Paths.get("a/b/assembly/assembly1.fits"), Annex),
  (Paths.get("a/b/c/assembly/assembly2.fits"), Normal),
  (Paths.get("testing/test.conf"), Normal)
)

// create config files at those paths
paths map { case (path, fileType) =>
  val createF = async {
    await(
      adminApi.create(path, ConfigData.fromString(defaultStrConf), Annex == fileType, "initial commit")
    )
  }
}

val responsesF = async {
  // retrieve list of all files;
  val allFilesF = await(adminApi.list()).map(info => info.path).toSet

  // retrieve list of files based on type;
  val allAnnexFilesF  = await(adminApi.list(Some(Annex))).map(info => info.path).toSet
  val allNormalFilesF = await(adminApi.list(Some(FileType.Normal))).map(info => info.path).toSet

  // retrieve list using pattern;
  val confFilesByPatternF = await(adminApi.list(None, Some(".*.conf"))).map(info => info.path.toString).toSet

  // retrieve list using pattern and file type;
  val allNormalConfFilesF = await(adminApi.list(Some(FileType.Normal), Some(".*.conf"))).map(info => info.path.toString).toSet

  val testConfF          = await(adminApi.list(Some(FileType.Normal), Some("test.*"))).map(info => info.path.toString).toSet
  val allAnnexConfFilesF = await(adminApi.list(Some(Annex), Some("a/c.*"))).map(info => info.path.toString).toSet

Java

copysourcePath trombonePath = Paths.get("a/c/trombone.conf");
Path hcdPath = Paths.get("a/b/c/hcd/hcd.conf");
Path fits1Path = Paths.get("a/b/assembly/assembly1.fits");
Path fits2Path = Paths.get("a/b/c/assembly/assembly2.fits");
Path testConfPath = Paths.get("testing/test.conf");

String comment = "initial commit";

//create files
ConfigId tromboneId = adminApi.create(trombonePath, ConfigData.fromString(defaultStrConf), true, comment).get();
ConfigId hcdId = adminApi.create(hcdPath, ConfigData.fromString(defaultStrConf), false, comment).get();
ConfigId fits1Id = adminApi.create(fits1Path, ConfigData.fromString(defaultStrConf), true, comment).get();
ConfigId fits2Id = adminApi.create(fits2Path, ConfigData.fromString(defaultStrConf), false, comment).get();
ConfigId testId = adminApi.create(testConfPath, ConfigData.fromString(defaultStrConf), true, comment).get();
//adminApi.list().get() = Set.of(tromboneId, hcdId, fits1Id, fits2Id, testId))
//adminApi.list(JFileType.Annex).get() = Set.of(tromboneId, fits1Id, testId)
//adminApi.list(JFileType.Normal).get() = Set.of(hcdId, fits2Id)
//adminApi.list(".*.conf").get() = Set.of(tromboneId, hcdId, testId)
//adminApi.list(JFileType.Annex, ".*.conf").get() = Set.of(tromboneId, testId)
//adminApi.list(JFileType.Annex, "a/c.*").get() = Set.of(tromboneId)
//adminApi.list(JFileType.Annex, "test.*").get() = Set.of(testId)

history

Returns the history of revisions of the file at the given path for a range of period specified by from and to. The size of the list can be restricted using maxResults.

Scala

copysourceval assertionF = async {
  val filePath = Paths.get("/a/test.conf")
  val id0      = await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "first commit"))

  // override the contents twice
  val tBeginUpdate = Instant.now()
  val id1          = await(adminApi.update(filePath, ConfigData.fromString("changing contents"), "second commit"))
  val id2          = await(adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third commit"))
  val tEndUpdate   = Instant.now()

  // full file history
  val fullHistory = await(adminApi.history(filePath))

  // drop initial revision and take only update revisions
  val revisionsBetweenF = await(adminApi.history(filePath, tBeginUpdate, tEndUpdate)).map(_.id)

  // take last two revisions
  val last2RevisionsF = await(adminApi.history(filePath, maxResults = 2)).map(_.id)

Java

copysourcePath filePath = Paths.get("/a/test.conf");
ConfigId id0 = adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "first commit").get();

//override the contents twice
Instant tBeginUpdate = Instant.now();
ConfigId id1 = adminApi.update(filePath, ConfigData.fromString("changing contents"), "second commit").get();
ConfigId id2 = adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third commit").get();
Instant tEndUpdate = Instant.now();
//adminApi.history(filePath).get().stream().map(ConfigFileRevision::id) = List.of(id2, id1, id0)
//adminApi.history(filePath).get().stream().map(ConfigFileRevision::comment) = List.of("third commit", "second commit", "first commit")
//adminApi.history(filePath, tBeginUpdate, tEndUpdate).get() = List.of(id2, id1)
//adminApi.history(filePath, 2).get().stream().map(ConfigFileRevision::id) = List.of(id2, id1)
//full file history

Managing active versions

Following API functions are available to manage the active version of a config file. In its lifetime, a config file undergoes many revisions. An active version is a specific revision from a file’s history and it is set by administrators.

• historyActive : Returns the history of active revisions of the file at the given path for a range of period specified by from and to. The size of the list can be restricted using maxResults.
• setActiveVersion : Sets the “active version” to be the version provided for the file at the given path. If this method is never called in a config’s lifetime, the active version will always be the version returned by create function.
• resetActiveVersion : Resets the “active version” of the file at the given path to the latest version.
• getActiveVersion : Returns the revision ID which represents the “active version” of the file at the given path.
• getActiveByTime : Returns the content of active version of the file existed at given instant

Scala

copysourceval assertionF = async {
  val tBegin   = Instant.now()
  val filePath = Paths.get("/a/test.conf")
  // create will make the 1st revision active with a default comment
  val id1      = await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "first"))
  val activeId = await(adminApi.historyActive(filePath)).map(_.id)

  // ensure active version is set
  val activeVersionF = await(adminApi.getActiveVersion(filePath)).get

  // override the contents four times
  await(adminApi.update(filePath, ConfigData.fromString("changing contents"), "second"))
  val id3 = await(adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third"))
  val id4 = await(adminApi.update(filePath, ConfigData.fromString("final contents"), "fourth"))
  val id5 = await(adminApi.update(filePath, ConfigData.fromString("final final contents"), "fifth"))

  // update doesn't change the active revision
  val idList = await(adminApi.historyActive(filePath)).map(_.id)
  // play with active version
  await(adminApi.setActiveVersion(filePath, id3, s"$id3 active"))
  await(adminApi.setActiveVersion(filePath, id4, s"$id4 active"))
  val id4F = await(adminApi.getActiveVersion(filePath)).get
  val tEnd = Instant.now()
  // reset active version to latest
  await(adminApi.resetActiveVersion(filePath, "latest active"))
  val id5F = await(adminApi.getActiveVersion(filePath)).get

  // finally set initial version as active
  await(adminApi.setActiveVersion(filePath, id1, s"$id1 active"))
  val id1F = await(adminApi.getActiveVersion(filePath)).get

  //  get full history
  val fullHistory = await(adminApi.historyActive(filePath))

  // drop initial revision and take only update revisions
  val fragmentedHistory = await(adminApi.historyActive(filePath, tBegin, tEnd))

  // take last three revisions
  val last3Revisions = await(adminApi.historyActive(filePath, maxResults = 3)).map(_.id)

  // get contents of active version at a specified instance
  val initialContents       = await(adminApi.getActiveByTime(filePath, tBegin)).get
  val initialContentsParseF = await(initialContents.toStringF(actorSystem))

Java

copysourceInstant tBegin = Instant.now();
Path filePath = Paths.get("/a/test.conf");

//create will make the 1st revision active with a default comment
ConfigId id1 = adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "first commit").get();
//adminApi.historyActive(filePath).get() = List.of(id1)
List<ConfigFileRevision> configFileRevisions = adminApi.historyActive(filePath).get();
//configFileRevisions = List.of(id1)
//ensure active version is set
Optional<ConfigId> configId1 = adminApi.getActiveVersion(filePath).get();
//configId1 = id1

//override the contents four times
adminApi.update(filePath, ConfigData.fromString("changing contents"), "second").get();
ConfigId id3 = adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third").get();
ConfigId id4 = adminApi.update(filePath, ConfigData.fromString("final contents"), "fourth").get();
ConfigId id5 = adminApi.update(filePath, ConfigData.fromString("final final contents"), "fifth").get();

//update doesn't change the active revision
Optional<ConfigId> configIdNow = adminApi.getActiveVersion(filePath).get();
//configIdNow = id1

//play with active version
adminApi.setActiveVersion(filePath, id3, "id3 active").get();
adminApi.setActiveVersion(filePath, id4, "id4 active").get();
Optional<ConfigId> configId4 = adminApi.getActiveVersion(filePath).get();

Instant tEnd = Instant.now();

//reset active version to latest
adminApi.resetActiveVersion(filePath, "latest active").get();
Optional<ConfigId> configId5 = adminApi.getActiveVersion(filePath).get();
//configId5 = id5
//finally set initial version as active
adminApi.setActiveVersion(filePath, id1, "id1 active").get();
Optional<ConfigId> configIdInitial = adminApi.getActiveVersion(filePath).get();
//configIdInitial = id1

//validate full history
List<ConfigFileRevision> fullHistory = adminApi.historyActive(filePath).get();
//fullHistory.stream().map(ConfigFileRevision::id) = List.of(id1, id5, id4, id3, id1)
//fullHistory.stream().map(ConfigFileRevision::comment) = List.of("id1 active", "latest active", "id4 active", "id3 active",
//                "initializing active file with the first version")


//drop initial revision and take only update revisions
List<ConfigFileRevision> fragmentedHistory = adminApi.historyActive(filePath, tBegin, tEnd).get();
//fragmentedHistory.size() = 3

//take last three revisions
CompletableFuture<List<ConfigFileRevision>> maxHistory = adminApi.historyActive(filePath, 3);
//maxHistory.get() = List.of(id1, id5, id4)


//get contents of active version at a specified instance
String initialContents = adminApi.getActiveByTime(filePath, tBegin).get().orElseThrow().toJStringF(actorSystem).get();
//initialContents = defaultStrConf

getMetaData

Used to get metadata information about the Config Service. It includes:

• repository directory
• annex directory
• min annex file size
• max config file size

Scala

copysourceval assertF = async {
  val metaData: ConfigMetadata = await(adminApi.getMetadata)
  //  metaData.repoPath returns repository path

Java

copysourceConfigMetadata metadata = adminApi.getMetadata().get();
//repository path must not be empty
//metadata.repoPath() != ""

Admin Protected Routes

The following Config Server routes are Admin Protected. To use these routes, the user must be authenticated and authorized.

• create
• update
• delete
• setActiveVersion
• resetActiveVersion

csw-config-client provides factory to create admin config service which allows access to these protected routes. This requires you to implement TokenFactory interface. Currently csw-config-cli is the only user of config service admin api. Refer to CliTokenFactory which implements TokenFactory interface.

Technical Description

See Configuration Service Technical Description.

Source code for examples

• Scala Example
• Java Example