Configuration Service
The Configuration Service provides a centralized persistent store for any configuration file used in the TMT Software System. All versions of configuration files are retained, providing a historical record of each configuration file.
Note that in order to use the APIs described here, the Location Service (csw-location-server) and Configuration Service Server needs to be running somewhere in the local network and the necessary configuration, environment variables or system properties should be defined to point to the correct host and port number(s) for the Location Service nodes.
This service will be part of the observatory cluster and exposes Rest endpoints that can be accessed over HTTP. Component developers can use the csw-config-client
library in their code. The library wraps the low level communication with Configuration Service Server and exposes simple to use methods to access and manage configuration files.
Dependencies
To use the Configuration Service without using the framework, add this to your build.sbt
file:
- sbt
-
libraryDependencies += "com.github.tmtsoftware.csw" %% "csw-config-client" % "5.0.1"
Rules and Checks
- The config file path must not contain
!#<>$%&'@^``~+,;=
orany whitespace character
- If the input file is > 10MB or has lot of non ASCII characters, then for optimization, server will archive it in
annex
store. - Large and binary files can be forced to go to the ‘annex’ store by using a
annex=true
flag in thecreate
operation. - API functions accept date-time values in UTC timezone. (e.g. 2017-05-17T08:00:24.246Z)
Model Classes
- ConfigData : Represents the contents of the files being managed. It wraps a stream of ByteString.
- ConfigFileInfo : Represents information about a config file stored in the Config Service.
- ConfigFileRevision : Represents information about a specific version of a config file.
- ConfigId : Represents an identifier associated with a revision of a configuration file, often generated by
create
orupdate
methods. - ConfigMetadata : Represents metadata information about the Config Server.
- FileType : Represents the type of storage for a configuration file. Currently two types are supported
Normal
(small, text files) andAnnex
(Large, Binary files).
API Flavors
The Configuration Service is used to provide the runtime settings for components. When a component is started, it will use a limited “clientAPI” to obtain the “active” configuration from the Configuration Service, and use those settings for its execution.
To change the active configuration, an administrative tool with access to the full “admin API” must be used. These tools would have the ability to create, delete, and update configurations, as well as retrieve past configurations and their history. Any time a new configuration is to be used by a component, the user must use one of these tools (via CLI, perhaps) to set the active configuration for a component. Since a history of active configurations is maintained by the service, the settings of each component each time it is run can be retrieved, and the system configuration at any moment can be recreated.
Some of the methods provided by the admin API, the methods that change the Config Server content, are protected so that a record can be kept about who is making the modifications. This is done by using a token obtained from the Authentication and Authorization Service. See Admin Protected Routes for more information.
- clientAPI : Must be used in Assembly and HCD components. Available functions are:
{exists | getActive}
- adminAPI : Full functionality exposed by Configuration Service Server is available with this API. Expected to be used administrators. Available functions are:
{create | update | getById | getLatest | getByTime | delete | list | history | historyActive | setActiveVersion | resetActiveVersion | getActiveVersion | getActiveByTime | getMetadata | exists | getActive}
Accessing clientAPI and adminAPI
The ConfigClientFactory
exposes functions to get the clientAPI and adminAPI. Both the functions require the Location Service instance which is used to resolve the ConfigServer
.
- Scala
-
source
// config client API val clientApi: ConfigClientService = ConfigClientFactory.clientApi(actorSystem, locationService) // config admin API val adminApi: ConfigService = ConfigClientFactory.adminApi(actorSystem, locationService, factory)
- Java
-
source
//config client API final IConfigClientService clientApi = JConfigClientFactory.clientApi(actorSystem, clientLocationService); //config admin API final IConfigService adminApi = JConfigClientFactory.adminApi(actorSystem, clientLocationService, mocks.factory());
Creating adminAPI
requires instance of TokenFactory
. TokenFactory
has a getToken
method which returns a raw access token string which is used by the config client to provide access to the Config Server. For more details, refer to bottom section on Admin Protected Routes
exists
This function checks if the file exists at specified path in the repository. It returns Future of a Boolean, whether the file exists or not.
- Scala
-
source
// construct the path val filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf") val doneF = async { // create file using admin API await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "First commit")) // check if file exists with config service val exists: Boolean = await(clientApi.exists(filePath)) // exists returns true/false
- Java
-
source
Path filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf"); // create file using admin API adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "commit config file").get(); Boolean exists = clientApi.exists(filePath).get();
getActive
This function retrieves the currently active file for a given path from config service. It returns a Future of Option of ConfigData.
- Scala
-
source
val defaultStrConf: String = "foo { bar { baz : 1234 } }" val doneF = async { // construct the path val filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf") await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "First commit")) val activeFile: Option[ConfigData] = await(clientApi.getActive(filePath)) // activeFile.get returns content of defaultStrConf
- Java
-
source
final String defaultStrConf = "foo { bar { baz : 1234 } }"; // construct the path Path filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf"); adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "First commit").get(); ConfigData activeFile = clientApi.getActive(filePath).get().orElseThrow(); String value = activeFile.toJConfigObject(actorSystem).get().getString("foo.bar.baz"); // value = 1234
create
Takes input ConfigData and creates the configuration in the repository at a specified path
- Scala
-
source
async { // construct ConfigData from String containing ASCII text val configString: String = """ // Name: ComponentType ConnectionType { name: lgsTromboneHCD type: Hcd connectionType: [akka] } """.stripMargin val config1: ConfigData = ConfigData.fromString(configString) // construct ConfigData from a local file containing binary data val srcFilePath = ResourceReader.copyToTmp("/smallBinary.bin") val config2: ConfigData = ConfigData.fromPath(srcFilePath) // construct ConfigData from Array[Byte] by reading a local file val stream: InputStream = getClass.getClassLoader.getResourceAsStream("smallBinary.bin") def byteArray: Array[Byte] = LazyList.continually(stream.read).takeWhile(_ != -1).map(_.toByte).toArray val config3 = ConfigData.fromBytes(byteArray) // store the config, at a specified path as normal text file val id1: ConfigId = await(adminApi.create(Paths.get("/hcd/trombone/overnight.conf"), config1, annex = false, "review done")) // store the config, at a specified path as a binary file in annex store val id2: ConfigId = await(adminApi.create(Paths.get("/hcd/trombone/firmware.bin"), config2, annex = true, "smoke test done")) // store the config, at a specified path as a binary file in annex store val id3: ConfigId = await(adminApi.create(Paths.get("/hcd/trombone/debug.bin"), config3, annex = true, "new file from vendor")) // id1 returns ConfigId(1) // id2 returns ConfigId(3) // id3 returns ConfigId(5)
- Java
-
source
//construct ConfigData from String containing ASCII text String configString = "axisName11111 = tromboneAxis\naxisName22222 = tromboneAxis2\naxisName3 = tromboneAxis3333"; ConfigData config1 = ConfigData.fromString(configString); //construct ConfigData from a local file containing binary data URI srcFilePath = getClass().getClassLoader().getResource("smallBinary.bin").toURI(); ConfigData config2 = ConfigData.fromPath(Paths.get(srcFilePath)); ConfigId id1 = adminApi.create(Paths.get("/hcd/trombone/overnight.conf"), config1, false, "review done").get(); ConfigId id2 = adminApi.create(Paths.get("/hcd/trombone/firmware.bin"), config2, true, "smoke test done").get(); //id1 = 1 //id2 = 3
update
Takes input ConfigData and overwrites the configuration specified in the repository
- Scala
-
source
val futU = async { val destPath = Paths.get("/hcd/trombone/debug.bin") val newId = await( adminApi .update(destPath, ConfigData.fromString(defaultStrConf), comment = "debug statements") ) // newId returns ConfigId(7)
- Java
-
source
Path destPath = Paths.get("/hcd/trombone/overnight.conf"); ConfigId newId = adminApi.update(destPath, ConfigData.fromString(defaultStrConf), "added debug statements").get(); //newId = 5
delete
Deletes a file located at specified path in the repository
- Scala
-
source
val futD = async { val unwantedFilePath = Paths.get("/hcd/trombone/debug.bin") await(adminApi.delete(unwantedFilePath, "no longer needed")) // validates the file is deleted val maybeConfigData = await(adminApi.getLatest(unwantedFilePath)) // maybeConfigData returns None
- Java
-
source
Path unwantedFilePath = Paths.get("/hcd/trombone/overnight.conf"); adminApi.delete(unwantedFilePath, "no longer needed").get(); Optional<ConfigData> expected = adminApi.getLatest(unwantedFilePath).get(); // expected = Optional.empty()
getById
Returns the file at a given path and matching revision Id
- Scala
-
source
val doneF = async { // create a file using API first val filePath = Paths.get("/tmt/trmobone/assembly/hcd.conf") val id: ConfigId = await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "First commit")) // validate val actualData = await(adminApi.getById(filePath, id)).get // actualData returns defaultStrConf
- Java
-
source
Path filePath = Paths.get("/tmt/trombone/assembly/hcd.conf"); ConfigId id = adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "First commit").get(); ConfigData actualData = adminApi.getById(filePath, id).get().orElseThrow(); // actualData.toJStringF(actorSystem).get() = defaultStrConf
getLatest
Returns the latest version of the file stored at the given path.
- Scala
-
source
val assertionF: Future[Assertion] = async { // create a file val filePath = Paths.get("/test.conf") await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "initial configuration")) // override the contents val newContent = "I changed the contents!!!" await(adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!")) // get the latest file val newConfigData = await(adminApi.getLatest(filePath)).get // validate // newConfigData returns newContent
- Java
-
source
//create a file Path filePath = Paths.get("/test.conf"); adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "initial configuration").get(); //override the contents String newContent = "I changed the contents!!!"; adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!").get(); //get the latest file ConfigData newConfigData = adminApi.getLatest(filePath).get().orElseThrow(); String filePathContent = newConfigData.toJStringF(actorSystem).get(); //filePathContent = newContent
getByTime
Gets the file at the given path as it existed at a given time-instance. Note:
- If time-instance is before the file was created, the initial version is returned.
- If time-instance is after the last change, the most recent version is returned.
- Scala
-
source
val assertionF = async { val tInitial = Instant.MIN // create a file val filePath = Paths.get("/a/b/c/test.conf") await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "initial configuration")) // override the contents val newContent = "I changed the contents!!!" await(adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!")) val initialData: ConfigData = await(adminApi.getByTime(filePath, tInitial)).get // initialData returns defaultStrConf val latestData = await(adminApi.getByTime(filePath, Instant.now())).get // latestData returns defaultStrConf
- Java
-
source
Instant tInitial = Instant.now(); //create a file Path filePath = Paths.get("/test.conf"); adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "initial configuration").get(); //override the contents String newContent = "I changed the contents!!!"; adminApi.update(filePath, ConfigData.fromString(newContent), "changed!!").get(); ConfigData initialData = adminApi.getByTime(filePath, tInitial).get().orElseThrow(); //initialData.toJStringF(actorSystem).get() = defaultStrConf ConfigData latestData = adminApi.getByTime(filePath, Instant.now()).get().orElseThrow(); //latestData.toJStringF(actorSystem).get() = newContent
list
For a given FileType (Annex or Normal) and an optional pattern string, it will list all files whose path matches the given pattern. Some pattern examples are: “/path/hcd/*.*”, “a/b/c/d.*”, “.*.conf”, “.*hcd.*”
- Scala
-
source
// Here's a list of tuples containing FilePath and FileTyepe(Annex / Normal) val paths: List[(Path, FileType)] = List[(Path, FileType)]( (Paths.get("a/c/trombone.conf"), Annex), (Paths.get("a/b/c/hcd/hcd.conf"), Normal), (Paths.get("a/b/assembly/assembly1.fits"), Annex), (Paths.get("a/b/c/assembly/assembly2.fits"), Normal), (Paths.get("testing/test.conf"), Normal) ) // create config files at those paths paths map { case (path, fileType) => val createF = async { await( adminApi.create(path, ConfigData.fromString(defaultStrConf), Annex == fileType, "initial commit") ) } } val responsesF = async { // retrieve list of all files; val allFilesF = await(adminApi.list()).map(info => info.path).toSet // retrieve list of files based on type; val allAnnexFilesF = await(adminApi.list(Some(Annex))).map(info => info.path).toSet val allNormalFilesF = await(adminApi.list(Some(FileType.Normal))).map(info => info.path).toSet // retrieve list using pattern; val confFilesByPatternF = await(adminApi.list(None, Some(".*.conf"))).map(info => info.path.toString).toSet // retrieve list using pattern and file type; val allNormalConfFilesF = await(adminApi.list(Some(FileType.Normal), Some(".*.conf"))).map(info => info.path.toString).toSet val testConfF = await(adminApi.list(Some(FileType.Normal), Some("test.*"))).map(info => info.path.toString).toSet val allAnnexConfFilesF = await(adminApi.list(Some(Annex), Some("a/c.*"))).map(info => info.path.toString).toSet
- Java
-
source
Path trombonePath = Paths.get("a/c/trombone.conf"); Path hcdPath = Paths.get("a/b/c/hcd/hcd.conf"); Path fits1Path = Paths.get("a/b/assembly/assembly1.fits"); Path fits2Path = Paths.get("a/b/c/assembly/assembly2.fits"); Path testConfPath = Paths.get("testing/test.conf"); String comment = "initial commit"; //create files ConfigId tromboneId = adminApi.create(trombonePath, ConfigData.fromString(defaultStrConf), true, comment).get(); ConfigId hcdId = adminApi.create(hcdPath, ConfigData.fromString(defaultStrConf), false, comment).get(); ConfigId fits1Id = adminApi.create(fits1Path, ConfigData.fromString(defaultStrConf), true, comment).get(); ConfigId fits2Id = adminApi.create(fits2Path, ConfigData.fromString(defaultStrConf), false, comment).get(); ConfigId testId = adminApi.create(testConfPath, ConfigData.fromString(defaultStrConf), true, comment).get(); //adminApi.list().get() = Set.of(tromboneId, hcdId, fits1Id, fits2Id, testId)) //adminApi.list(JFileType.Annex).get() = Set.of(tromboneId, fits1Id, testId) //adminApi.list(JFileType.Normal).get() = Set.of(hcdId, fits2Id) //adminApi.list(".*.conf").get() = Set.of(tromboneId, hcdId, testId) //adminApi.list(JFileType.Annex, ".*.conf").get() = Set.of(tromboneId, testId) //adminApi.list(JFileType.Annex, "a/c.*").get() = Set.of(tromboneId) //adminApi.list(JFileType.Annex, "test.*").get() = Set.of(testId)
history
Returns the history of revisions of the file at the given path for a range of period specified by from
and to
. The size of the list can be restricted using maxResults
.
- Scala
-
source
val assertionF = async { val filePath = Paths.get("/a/test.conf") val id0 = await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "first commit")) // override the contents twice val tBeginUpdate = Instant.now() val id1 = await(adminApi.update(filePath, ConfigData.fromString("changing contents"), "second commit")) val id2 = await(adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third commit")) val tEndUpdate = Instant.now() // full file history val fullHistory = await(adminApi.history(filePath)) // drop initial revision and take only update revisions val revisionsBetweenF = await(adminApi.history(filePath, tBeginUpdate, tEndUpdate)).map(_.id) // take last two revisions val last2RevisionsF = await(adminApi.history(filePath, maxResults = 2)).map(_.id)
- Java
-
source
Path filePath = Paths.get("/a/test.conf"); ConfigId id0 = adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "first commit").get(); //override the contents twice Instant tBeginUpdate = Instant.now(); ConfigId id1 = adminApi.update(filePath, ConfigData.fromString("changing contents"), "second commit").get(); ConfigId id2 = adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third commit").get(); Instant tEndUpdate = Instant.now(); //adminApi.history(filePath).get().stream().map(ConfigFileRevision::id) = List.of(id2, id1, id0) //adminApi.history(filePath).get().stream().map(ConfigFileRevision::comment) = List.of("third commit", "second commit", "first commit") //adminApi.history(filePath, tBeginUpdate, tEndUpdate).get() = List.of(id2, id1) //adminApi.history(filePath, 2).get().stream().map(ConfigFileRevision::id) = List.of(id2, id1) //full file history
Managing active versions
Following API functions are available to manage the active version of a config file. In its lifetime, a config file undergoes many revisions. An active version is a specific revision from a file’s history and it is set by administrators.
- historyActive : Returns the history of active revisions of the file at the given path for a range of period specified by
from
andto
. The size of the list can be restricted usingmaxResults
. - setActiveVersion : Sets the “active version” to be the version provided for the file at the given path. If this method is never called in a config’s lifetime, the active version will always be the version returned by
create
function. - resetActiveVersion : Resets the “active version” of the file at the given path to the latest version.
- getActiveVersion : Returns the revision ID which represents the “active version” of the file at the given path.
- getActiveByTime : Returns the content of active version of the file existed at given instant
- Scala
-
source
val assertionF = async { val tBegin = Instant.now() val filePath = Paths.get("/a/test.conf") // create will make the 1st revision active with a default comment val id1 = await(adminApi.create(filePath, ConfigData.fromString(defaultStrConf), annex = false, "first")) val activeId = await(adminApi.historyActive(filePath)).map(_.id) // ensure active version is set val activeVersionF = await(adminApi.getActiveVersion(filePath)).get // override the contents four times await(adminApi.update(filePath, ConfigData.fromString("changing contents"), "second")) val id3 = await(adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third")) val id4 = await(adminApi.update(filePath, ConfigData.fromString("final contents"), "fourth")) val id5 = await(adminApi.update(filePath, ConfigData.fromString("final final contents"), "fifth")) // update doesn't change the active revision val idList = await(adminApi.historyActive(filePath)).map(_.id) // play with active version await(adminApi.setActiveVersion(filePath, id3, s"$id3 active")) await(adminApi.setActiveVersion(filePath, id4, s"$id4 active")) val id4F = await(adminApi.getActiveVersion(filePath)).get val tEnd = Instant.now() // reset active version to latest await(adminApi.resetActiveVersion(filePath, "latest active")) val id5F = await(adminApi.getActiveVersion(filePath)).get // finally set initial version as active await(adminApi.setActiveVersion(filePath, id1, s"$id1 active")) val id1F = await(adminApi.getActiveVersion(filePath)).get // get full history val fullHistory = await(adminApi.historyActive(filePath)) // drop initial revision and take only update revisions val fragmentedHistory = await(adminApi.historyActive(filePath, tBegin, tEnd)) // take last three revisions val last3Revisions = await(adminApi.historyActive(filePath, maxResults = 3)).map(_.id) // get contents of active version at a specified instance val initialContents = await(adminApi.getActiveByTime(filePath, tBegin)).get val initialContentsParseF = await(initialContents.toStringF(actorSystem))
- Java
-
source
Instant tBegin = Instant.now(); Path filePath = Paths.get("/a/test.conf"); //create will make the 1st revision active with a default comment ConfigId id1 = adminApi.create(filePath, ConfigData.fromString(defaultStrConf), false, "first commit").get(); //adminApi.historyActive(filePath).get() = List.of(id1) List<ConfigFileRevision> configFileRevisions = adminApi.historyActive(filePath).get(); //configFileRevisions = List.of(id1) //ensure active version is set Optional<ConfigId> configId1 = adminApi.getActiveVersion(filePath).get(); //configId1 = id1 //override the contents four times adminApi.update(filePath, ConfigData.fromString("changing contents"), "second").get(); ConfigId id3 = adminApi.update(filePath, ConfigData.fromString("changing contents again"), "third").get(); ConfigId id4 = adminApi.update(filePath, ConfigData.fromString("final contents"), "fourth").get(); ConfigId id5 = adminApi.update(filePath, ConfigData.fromString("final final contents"), "fifth").get(); //update doesn't change the active revision Optional<ConfigId> configIdNow = adminApi.getActiveVersion(filePath).get(); //configIdNow = id1 //play with active version adminApi.setActiveVersion(filePath, id3, "id3 active").get(); adminApi.setActiveVersion(filePath, id4, "id4 active").get(); Optional<ConfigId> configId4 = adminApi.getActiveVersion(filePath).get(); Instant tEnd = Instant.now(); //reset active version to latest adminApi.resetActiveVersion(filePath, "latest active").get(); Optional<ConfigId> configId5 = adminApi.getActiveVersion(filePath).get(); //configId5 = id5 //finally set initial version as active adminApi.setActiveVersion(filePath, id1, "id1 active").get(); Optional<ConfigId> configIdInitial = adminApi.getActiveVersion(filePath).get(); //configIdInitial = id1 //validate full history List<ConfigFileRevision> fullHistory = adminApi.historyActive(filePath).get(); //fullHistory.stream().map(ConfigFileRevision::id) = List.of(id1, id5, id4, id3, id1) //fullHistory.stream().map(ConfigFileRevision::comment) = List.of("id1 active", "latest active", "id4 active", "id3 active", // "initializing active file with the first version") //drop initial revision and take only update revisions List<ConfigFileRevision> fragmentedHistory = adminApi.historyActive(filePath, tBegin, tEnd).get(); //fragmentedHistory.size() = 3 //take last three revisions CompletableFuture<List<ConfigFileRevision>> maxHistory = adminApi.historyActive(filePath, 3); //maxHistory.get() = List.of(id1, id5, id4) //get contents of active version at a specified instance String initialContents = adminApi.getActiveByTime(filePath, tBegin).get().orElseThrow().toJStringF(actorSystem).get(); //initialContents = defaultStrConf
getMetaData
Used to get metadata information about the Config Service. It includes:
- repository directory
- annex directory
- min annex file size
- max config file size
- Scala
-
source
val assertF = async { val metaData: ConfigMetadata = await(adminApi.getMetadata) // metaData.repoPath returns repository path
- Java
-
source
ConfigMetadata metadata = adminApi.getMetadata().get(); //repository path must not be empty //metadata.repoPath() != ""
Admin Protected Routes
The following Config Server routes are Admin Protected
. To use these routes, the user must be authenticated and authorized.
- create
- update
- delete
- setActiveVersion
- resetActiveVersion
csw-config-client
provides factory to create admin config service which allows access to these protected routes. This requires you to implement TokenFactory interface. Currently csw-config-cli
is the only user of config service admin api. Refer to CliTokenFactory which implements TokenFactory
interface.
Refer to csw-aas docs to know more about how to authenticate and authorize with AAS and get an access token.
Technical Description
See Configuration Service Technical Description.