diff --git a/README.md b/README.md index 1799999b..7b68c6be 100644 --- a/README.md +++ b/README.md @@ -1,69 +1,8 @@ -![GitHub release](https://img.shields.io/github/release/marklogic/ml-app-deployer.svg) -![GitHub last commit](https://img.shields.io/github/last-commit/marklogic/ml-app-deployer.svg) -[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -[![Known Vulnerabilities](https://snyk.io/test/github/marklogic/ml-app-deployer/badge.svg)](https://snyk.io/test/github/marklogic/ml-app-deployer) +## Archive notice -ml-app-deployer is a Java library that provides two capabilities: +This repository has been archived as the code has been migrated to the +[ml-gradle repository](https://github.com/marklogic/ml-gradle/tree/dev/ml-app-deployer) as part of the +ml-gradle 5.0.0 release. -1. A client library for the [MarkLogic Management REST API](http://docs.marklogic.com/REST/management) -1. A command-driven approach for deploying and undeploying an application to MarkLogic that depends on the management client library - -If you're just looking for a Java library for interacting with the Management REST API, you can certainly use ml-app-deployer. -The deployer/command library can be safely ignored if you don't need it. - -### What does ml-app-deployer depend on? - -ml-app-deployer depends on MarkLogic 10 and Java 1.8+. Earlier versions of MarkLogic may work, but due to improvements -and bug fixes in the MarkLogic Management REST API across versions 8, 9, and 10, it is recommended to use MarkLogic 10. - -Under the hood, it depends on Spring's [RestTemplate](http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html) for interacting with the Management REST API. -It also depends on [ml-javaclient-util](https://github.com/marklogic/ml-javaclient-util) for loading modules, which is done via the MarkLogic Client REST API. - -### How do I start using the client library? - -The general pattern for using the management client library is: - -1. Create an instance of [ManageConfig](https://github.com/marklogic/ml-app-deployer/blob/master/src/main/java/com/marklogic/mgmt/ManageConfig.java), which specifies connection information for the management REST API instance. -2. Create an instance of [ManageClient](https://github.com/marklogic/ml-app-deployer/blob/master/src/main/java/com/marklogic/mgmt/ManageClient.java) using ManageConfig. ManageClient simply wraps a RestTemplate with some convenience methods. -3. Using ManageClient, create a Manager class based on the management resource you want to configure. For example, to create or modify or delete a database, create a [DatabaseManager](https://github.com/marklogic/ml-app-deployer/blob/master/src/main/java/com/marklogic/mgmt/databases/DatabaseManager.java) to talk to the [database endpoints](http://docs.marklogic.com/REST/management/databases). - -Here's a brief example of what that looks like: - - ManageConfig config = new ManageConfig(); // defaults to localhost/8002/admin/admin - ManageClient client = new ManageClient(config); - DatabaseManager dbMgr = new DatabaseManager(client); - dbMgr.save("{\"database-name\":\"my-database\"}"); - -### How do I start using the deployer library? - -The main concept behind the deployer library is invoke a series of commands, where each command looks for one or more configuration files in a specific directory structure and then uses a Manager class in the client library to apply those configuration files as part of deploying an application. - -The best way to understand that directory is to look at the [sample-app application](https://github.com/marklogic/ml-app-deployer/tree/master/src/test/resources/sample-app/src/main/ml-config) that's used by the JUnit tests. The concept is fairly simple - within the ml-config directory, there's a directory for each of the top-level resources defined by the [Management API docs](http://docs.marklogic.com/REST/management). Thus, database config files are found under "databases", while scheduled task config files are found under "scheduled-tasks". Some directories have subdirectories based on how the Management API endpoints are defined - for example, the "security" directory has child directories of "amps", "roles", "users", and others based on the resources that comprise the "security" set of endpoints. - -The logic for when to look for files is encapsulated in Command objects. A deployment is performed by one or more Command objects. Thus, the general pattern for using the deployer library is: - -1. Create an instance of SimpleAppDeployer, which implements the AppDeployer interface -2. Set a list of commands on the SimpleAppDeployer instance -3. Call the "deploy" method to invoke each of the commands in a specific order - -Here's a brief example of what that looks like - note that we'll reuse our ManageClient from above, and we'll deploy an -application that needs to create a REST API server named "my-app" on port 8123 and create some users too - the config for both of those will be read from -files in the ml-config directory structure: - - ManageClient client = new ManageClient(); // defaults to localhost/8002/admin/admin - AdminManager manager = new AdminManager(); // used for restarting ML; defaults to localhost/8001/admin/admin - AppDeployer deployer = new SimpleAppDeployer(client, manager, - new DeployRestApiServersCommand(), new DeployUsersCommand()); - - // AppConfig contains all configuration about the application being deployed - AppConfig config = new AppConfig(); - config.setName("my-app"); - config.setRestPort(8123); - - // Calls each command, passing the AppConfig and ManageClient to each one - deployer.deploy(config); - - // do some other stuff... - - // Calls each command, giving each a chance to undo what it did before - deployer.undeploy(config); +The [Wiki pages in this repository](https://github.com/marklogic/ml-app-deployer/wiki) are still valid as +documentation but may soon be folded into the ml-gradle documentation. diff --git a/build.gradle b/build.gradle index e69f49a5..93f4b0c9 100644 --- a/build.gradle +++ b/build.gradle @@ -3,12 +3,12 @@ plugins { id "maven-publish" id "signing" id "com.github.jk1.dependency-license-report" version "1.17" - id "net.saliman.properties" version "1.5.1" + id "net.saliman.properties" version "1.5.2" id "io.snyk.gradle.plugin.snykplugin" version "0.4" } group = "com.marklogic" -version = "4.6.0" +version = "4.8.0" java { sourceCompatibility = 1.8 @@ -19,33 +19,31 @@ repositories { mavenLocal() mavenCentral() maven { - url "https://nexus.marklogic.com/repository/maven-snapshots/" + url "https://bed-artifactory.bedford.progress.com:443/artifactory/ml-maven-snapshots/" } } dependencies { - api 'com.marklogic:ml-javaclient-util:4.6.0' - api 'org.springframework:spring-web:5.3.29' - api 'com.fasterxml.jackson.core:jackson-databind:2.15.2' + api 'com.marklogic:ml-javaclient-util:4.8.0' + api 'org.springframework:spring-web:5.3.34' + api 'com.fasterxml.jackson.core:jackson-databind:2.15.3' implementation 'jaxen:jaxen:1.2.0' - // Forcing usage of 3.4.0 instead of 3.2.0 to address vulnerability - https://security.snyk.io/vuln/SNYK-JAVA-COMSQUAREUPOKIO-5820002 - implementation 'com.squareup.okio:okio:3.4.0' - implementation 'com.squareup.okhttp3:okhttp:4.11.0' + implementation 'com.squareup.okhttp3:okhttp:4.12.0' implementation 'io.github.rburgst:okhttp-digest:2.7' implementation 'org.apache.httpcomponents:httpclient:4.5.14' implementation 'org.jdom:jdom2:2.0.6.1' // Forcing httpclient to use this to address https://snyk.io/vuln/SNYK-JAVA-COMMONSCODEC-561518 - implementation 'commons-codec:commons-codec:1.15' + implementation 'commons-codec:commons-codec:1.16.1' // For EqualsBuilder; added in 3.8.1 to support detecting if a mimetype's properties have changed or not - implementation "org.apache.commons:commons-lang3:3.12.0" + implementation "org.apache.commons:commons-lang3:3.14.0" // For PreviewInterceptor; can be excluded if that feature is not used - implementation("com.flipkart.zjsonpatch:zjsonpatch:0.4.14") { + implementation("com.flipkart.zjsonpatch:zjsonpatch:0.4.16") { // Prefer the api version declared above exclude module: "jackson-databind" } @@ -57,31 +55,31 @@ dependencies { // Don't want to include this in the published jar, just the executable jar compileOnly "com.beust:jcommander:1.82" - compileOnly "ch.qos.logback:logback-classic:1.3.5" + compileOnly "ch.qos.logback:logback-classic:1.3.14" - testImplementation 'org.junit.jupiter:junit-jupiter:5.10.0' - testImplementation 'org.springframework:spring-test:5.3.29' - testImplementation 'commons-io:commons-io:2.11.0' + testImplementation 'org.junit.jupiter:junit-jupiter:5.10.2' + testImplementation 'org.springframework:spring-test:5.3.34' + testImplementation 'commons-io:commons-io:2.16.1' testImplementation 'xmlunit:xmlunit:1.6' // Forcing Spring to use logback for testing instead of commons-logging - testImplementation "ch.qos.logback:logback-classic:1.3.5" - testImplementation "org.slf4j:jcl-over-slf4j:1.7.36" - testImplementation "org.slf4j:slf4j-api:1.7.36" + testImplementation "ch.qos.logback:logback-classic:1.3.14" + testImplementation "org.slf4j:jcl-over-slf4j:2.0.13" + testImplementation "org.slf4j:slf4j-api:2.0.13" } // This ensures that Gradle includes in the published jar any non-java files under src/main/java sourceSets.main.resources.srcDir 'src/main/java' task sourcesJar(type: Jar, dependsOn: classes) { - classifier 'sources' + archiveClassifier = 'sources' from sourceSets.main.allSource // For unknown reasons, Gradle 7.1 (but not 6.x) is complaining that AbstractManager.java is a duplicate. duplicatesStrategy = "exclude" } task javadocJar(type: Jar, dependsOn: javadoc) { - classifier "javadoc" + archiveClassifier = "javadoc" from javadoc } javadoc.failOnError = false @@ -131,9 +129,9 @@ publishing { } } scm { - url = "git@github.com:marklogic-community/${project.name}.git" - connection = "scm:git@github.com:marklogic-community/${project.name}.git" - developerConnection = "scm:git@github.com:marklogic-community/${project.name}.git" + url = "git@github.com:marklogic/${project.name}.git" + connection = "scm:git@github.com:marklogic/${project.name}.git" + developerConnection = "scm:git@github.com:marklogic/${project.name}.git" } } from components.java diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index 1d4d3ea4..f20ef03b 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -3,4 +3,4 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-7.5.1-bin.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-8.5-bin.zip diff --git a/pom.xml b/pom.xml index 22830b64..b4110e59 100644 --- a/pom.xml +++ b/pom.xml @@ -12,7 +12,7 @@ It is not intended to be used to build this project. 4.0.0 com.marklogic ml-app-deployer - 4.6.0 + 4.8.0 com.marklogic:ml-app-deployer Java client for the MarkLogic REST Management API and for deploying applications to MarkLogic https://github.com/marklogic/ml-app-deployer @@ -40,19 +40,19 @@ It is not intended to be used to build this project. com.marklogic ml-javaclient-util - 4.6.0 + 4.8.0 compile org.springframework spring-web - 5.3.29 + 5.3.34 compile com.fasterxml.jackson.core jackson-databind - 2.15.2 + 2.15.3 compile @@ -76,19 +76,19 @@ It is not intended to be used to build this project. commons-codec commons-codec - 1.15 + 1.16.1 runtime org.apache.commons commons-lang3 - 3.12.0 + 3.14.0 runtime com.flipkart.zjsonpatch zjsonpatch - 0.4.14 + 0.4.16 runtime diff --git a/src/main/java/com/marklogic/appdeployer/AppConfig.java b/src/main/java/com/marklogic/appdeployer/AppConfig.java index fef56566..c32adaf8 100644 --- a/src/main/java/com/marklogic/appdeployer/AppConfig.java +++ b/src/main/java/com/marklogic/appdeployer/AppConfig.java @@ -121,6 +121,16 @@ public class AppConfig { private String restTrustManagementAlgorithm; private String restBasePath; + // Added in 4.7.0 + private String restKeyStorePath; + private String restKeyStorePassword; + private String restKeyStoreType; + private String restKeyStoreAlgorithm; + private String restTrustStorePath; + private String restTrustStorePassword; + private String restTrustStoreType; + private String restTrustStoreAlgorithm; + private Integer restPort = DEFAULT_PORT; private Integer testRestPort; private String testRestBasePath; @@ -143,6 +153,16 @@ public class AppConfig { private String appServicesTrustManagementAlgorithm; private String appServicesBasePath; + // Added in 4.7.0 + private String appServicesKeyStorePath; + private String appServicesKeyStorePassword; + private String appServicesKeyStoreType; + private String appServicesKeyStoreAlgorithm; + private String appServicesTrustStorePath; + private String appServicesTrustStorePassword; + private String appServicesTrustStoreType; + private String appServicesTrustStoreAlgorithm; + // These can all be set to override the default names that are generated off of the "name" attribute. private String groupName = DEFAULT_GROUP; private boolean noRestServer = false; @@ -416,6 +436,15 @@ public DatabaseClientConfig newRestDatabaseClientConfig(int port) { config.setCloudApiKey(cloudApiKey); config.setBasePath(restBasePath); + config.setKeyStorePath(restKeyStorePath); + config.setKeyStorePassword(restKeyStorePassword); + config.setKeyStoreType(restKeyStoreType); + config.setKeyStoreAlgorithm(restKeyStoreAlgorithm); + config.setTrustStorePath(restTrustStorePath); + config.setTrustStorePassword(restTrustStorePassword); + config.setTrustStoreType(restTrustStoreType); + config.setTrustStoreAlgorithm(restTrustStoreAlgorithm); + if (restUseDefaultKeystore) { config.setSslProtocol(StringUtils.hasText(restSslProtocol) ? restSslProtocol : SslUtil.DEFAULT_SSL_PROTOCOL); config.setTrustManagementAlgorithm(restTrustManagementAlgorithm); @@ -459,6 +488,15 @@ public DatabaseClient newAppServicesDatabaseClient(String databaseName) { config.setCloudApiKey(cloudApiKey); config.setBasePath(appServicesBasePath); + config.setKeyStorePath(appServicesKeyStorePath); + config.setKeyStorePassword(appServicesKeyStorePassword); + config.setKeyStoreType(appServicesKeyStoreType); + config.setKeyStoreAlgorithm(appServicesKeyStoreAlgorithm); + config.setTrustStorePath(appServicesTrustStorePath); + config.setTrustStorePassword(appServicesTrustStorePassword); + config.setTrustStoreType(appServicesTrustStoreType); + config.setTrustStoreAlgorithm(appServicesTrustStoreAlgorithm); + if (appServicesUseDefaultKeystore) { config.setSslProtocol(StringUtils.hasText(appServicesSslProtocol) ? appServicesSslProtocol : SslUtil.DEFAULT_SSL_PROTOCOL); config.setTrustManagementAlgorithm(appServicesTrustManagementAlgorithm); @@ -1556,4 +1594,292 @@ public boolean isCascadePermissions() { public void setCascadePermissions(boolean cascadePermissions) { this.cascadePermissions = cascadePermissions; } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestKeyStorePath() { + return restKeyStorePath; + } + + /** + * + * @param restKeyStorePath + * @since 4.7.0 + */ + public void setRestKeyStorePath(String restKeyStorePath) { + this.restKeyStorePath = restKeyStorePath; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestKeyStorePassword() { + return restKeyStorePassword; + } + + /** + * + * @param restKeyStorePassword + * @since 4.7.0 + */ + public void setRestKeyStorePassword(String restKeyStorePassword) { + this.restKeyStorePassword = restKeyStorePassword; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestKeyStoreType() { + return restKeyStoreType; + } + + /** + * + * @param restKeyStoreType + * @since 4.7.0 + */ + public void setRestKeyStoreType(String restKeyStoreType) { + this.restKeyStoreType = restKeyStoreType; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestKeyStoreAlgorithm() { + return restKeyStoreAlgorithm; + } + + /** + * + * @param restKeyStoreAlgorithm + * @since 4.7.0 + */ + public void setRestKeyStoreAlgorithm(String restKeyStoreAlgorithm) { + this.restKeyStoreAlgorithm = restKeyStoreAlgorithm; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestTrustStorePath() { + return restTrustStorePath; + } + + /** + * + * @param restTrustStorePath + * @since 4.7.0 + */ + public void setRestTrustStorePath(String restTrustStorePath) { + this.restTrustStorePath = restTrustStorePath; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestTrustStorePassword() { + return restTrustStorePassword; + } + + /** + * + * @param restTrustStorePassword + * @since 4.7.0 + */ + public void setRestTrustStorePassword(String restTrustStorePassword) { + this.restTrustStorePassword = restTrustStorePassword; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestTrustStoreType() { + return restTrustStoreType; + } + + /** + * + * @param restTrustStoreType + * @since 4.7.0 + */ + public void setRestTrustStoreType(String restTrustStoreType) { + this.restTrustStoreType = restTrustStoreType; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getRestTrustStoreAlgorithm() { + return restTrustStoreAlgorithm; + } + + /** + * + * @param restTrustStoreAlgorithm + * @since 4.7.0 + */ + public void setRestTrustStoreAlgorithm(String restTrustStoreAlgorithm) { + this.restTrustStoreAlgorithm = restTrustStoreAlgorithm; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesKeyStorePath() { + return appServicesKeyStorePath; + } + + /** + * + * @param appServicesKeyStorePath + * @since 4.7.0 + */ + public void setAppServicesKeyStorePath(String appServicesKeyStorePath) { + this.appServicesKeyStorePath = appServicesKeyStorePath; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesKeyStorePassword() { + return appServicesKeyStorePassword; + } + + /** + * + * @param appServicesKeyStorePassword + * @since 4.7.0 + */ + public void setAppServicesKeyStorePassword(String appServicesKeyStorePassword) { + this.appServicesKeyStorePassword = appServicesKeyStorePassword; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesKeyStoreType() { + return appServicesKeyStoreType; + } + + /** + * + * @param appServicesKeyStoreType + * @since 4.7.0 + */ + public void setAppServicesKeyStoreType(String appServicesKeyStoreType) { + this.appServicesKeyStoreType = appServicesKeyStoreType; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesKeyStoreAlgorithm() { + return appServicesKeyStoreAlgorithm; + } + + /** + * + * @param appServicesKeyStoreAlgorithm + * @since 4.7.0 + */ + public void setAppServicesKeyStoreAlgorithm(String appServicesKeyStoreAlgorithm) { + this.appServicesKeyStoreAlgorithm = appServicesKeyStoreAlgorithm; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesTrustStorePath() { + return appServicesTrustStorePath; + } + + /** + * + * @param appServicesTrustStorePath + * @since 4.7.0 + */ + public void setAppServicesTrustStorePath(String appServicesTrustStorePath) { + this.appServicesTrustStorePath = appServicesTrustStorePath; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesTrustStorePassword() { + return appServicesTrustStorePassword; + } + + /** + * + * @param appServicesTrustStorePassword + * @since 4.7.0 + */ + public void setAppServicesTrustStorePassword(String appServicesTrustStorePassword) { + this.appServicesTrustStorePassword = appServicesTrustStorePassword; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesTrustStoreType() { + return appServicesTrustStoreType; + } + + /** + * + * @param appServicesTrustStoreType + * @since 4.7.0 + */ + public void setAppServicesTrustStoreType(String appServicesTrustStoreType) { + this.appServicesTrustStoreType = appServicesTrustStoreType; + } + + /** + * + * @return + * @since 4.7.0 + */ + public String getAppServicesTrustStoreAlgorithm() { + return appServicesTrustStoreAlgorithm; + } + + /** + * + * @param appServicesTrustStoreAlgorithm + * @since 4.7.0 + */ + public void setAppServicesTrustStoreAlgorithm(String appServicesTrustStoreAlgorithm) { + this.appServicesTrustStoreAlgorithm = appServicesTrustStoreAlgorithm; + } } diff --git a/src/main/java/com/marklogic/appdeployer/ConfigDir.java b/src/main/java/com/marklogic/appdeployer/ConfigDir.java index aa7adf4a..3dac0b44 100644 --- a/src/main/java/com/marklogic/appdeployer/ConfigDir.java +++ b/src/main/java/com/marklogic/appdeployer/ConfigDir.java @@ -262,4 +262,8 @@ public File getProjectDir() { public File getSecureCredentialsDir() { return new File(getSecurityDir(), "secure-credentials"); } + + public File getCredentialsDir() { + return new File(getSecurityDir(), "credentials"); + } } diff --git a/src/main/java/com/marklogic/appdeployer/DefaultAppConfigFactory.java b/src/main/java/com/marklogic/appdeployer/DefaultAppConfigFactory.java index f0612338..ad836ae2 100644 --- a/src/main/java/com/marklogic/appdeployer/DefaultAppConfigFactory.java +++ b/src/main/java/com/marklogic/appdeployer/DefaultAppConfigFactory.java @@ -17,7 +17,6 @@ import com.marklogic.appdeployer.util.JavaClientUtil; import com.marklogic.client.DatabaseClient; -import com.marklogic.client.DatabaseClientFactory; import com.marklogic.client.ext.SecurityContextType; import com.marklogic.mgmt.util.PropertySource; import com.marklogic.mgmt.util.PropertySourceFactory; @@ -206,13 +205,52 @@ public void initialize() { config.setCloudApiKey(prop); }); + propertyConsumerMap.put("mlKeyStorePath", (config, prop) -> { + logger.info("REST and App-Services key store path: " + prop); + config.setRestKeyStorePath(prop); + config.setAppServicesKeyStorePath(prop); + }); + propertyConsumerMap.put("mlKeyStorePassword", (config, prop) -> { + config.setRestKeyStorePassword(prop); + config.setAppServicesKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlKeyStoreType", (config, prop) -> { + logger.info("REST and App-Services key store type: " + prop); + config.setRestKeyStoreType(prop); + config.setAppServicesKeyStoreType(prop); + }); + propertyConsumerMap.put("mlKeyStoreAlgorithm", (config, prop) -> { + logger.info("REST and App-Services key store algorithm: " + prop); + config.setRestKeyStoreAlgorithm(prop); + config.setAppServicesKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlTrustStorePath", (config, prop) -> { + logger.info("REST and App-Services trust store path: " + prop); + config.setRestTrustStorePath(prop); + config.setAppServicesTrustStorePath(prop); + }); + propertyConsumerMap.put("mlTrustStorePassword", (config, prop) -> { + config.setRestTrustStorePassword(prop); + config.setAppServicesTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlTrustStoreType", (config, prop) -> { + logger.info("REST and App-Services trust store type: " + prop); + config.setRestTrustStoreType(prop); + config.setAppServicesTrustStoreType(prop); + }); + propertyConsumerMap.put("mlTrustStoreAlgorithm", (config, prop) -> { + logger.info("REST and App-Services trust store algorithm: " + prop); + config.setRestTrustStoreAlgorithm(prop); + config.setAppServicesTrustStoreAlgorithm(prop); + }); + /** * Defaults to port 8000. In rare cases, the ML App-Services app server will have been changed to listen on a * different port, in which case you can set this to that port. */ propertyConsumerMap.put("mlAppServicesPort", (config, prop) -> { - logger.info("App services port: " + prop); - config.setAppServicesPort(Integer.parseInt(prop)); + logger.info("App services port: {}", prop); + config.setAppServicesPort(propertyToInteger("mlAppServicesPort", prop)); }); /** * The username and password for a ML user with the rest-admin role that is used for e.g. loading @@ -293,6 +331,37 @@ public void initialize() { config.setAppServicesBasePath(appServicesPath); }); + propertyConsumerMap.put("mlAppServicesKeyStorePath", (config, prop) -> { + logger.info("App-Services key store path: " + prop); + config.setAppServicesKeyStorePath(prop); + }); + propertyConsumerMap.put("mlAppServicesKeyStorePassword", (config, prop) -> { + config.setAppServicesKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlAppServicesKeyStoreType", (config, prop) -> { + logger.info("App-Services key store type: " + prop); + config.setAppServicesKeyStoreType(prop); + }); + propertyConsumerMap.put("mlAppServicesKeyStoreAlgorithm", (config, prop) -> { + logger.info("App-Services key store algorithm: " + prop); + config.setAppServicesKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlAppServicesTrustStorePath", (config, prop) -> { + logger.info("App-Services trust store path: " + prop); + config.setAppServicesTrustStorePath(prop); + }); + propertyConsumerMap.put("mlAppServicesTrustStorePassword", (config, prop) -> { + config.setAppServicesTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlAppServicesTrustStoreType", (config, prop) -> { + logger.info("App-Services trust store type: " + prop); + config.setAppServicesTrustStoreType(prop); + }); + propertyConsumerMap.put("mlAppServicesTrustStoreAlgorithm", (config, prop) -> { + logger.info("App-Services trust store algorithm: " + prop); + config.setAppServicesTrustStoreAlgorithm(prop); + }); + /** * Set this to true to prevent creating a REST API server by default. */ @@ -306,7 +375,7 @@ public void initialize() { */ propertyConsumerMap.put("mlRestPort", (config, prop) -> { logger.info("App REST port: " + prop); - config.setRestPort(Integer.parseInt(prop)); + config.setRestPort(propertyToInteger("mlRestPort", prop)); }); /** * The username and password for a ML user with the rest-admin role. This user is used for operations against the @@ -421,6 +490,37 @@ public void initialize() { config.setRestTrustManagementAlgorithm(prop); }); + propertyConsumerMap.put("mlRestKeyStorePath", (config, prop) -> { + logger.info("REST key store path: " + prop); + config.setRestKeyStorePath(prop); + }); + propertyConsumerMap.put("mlRestKeyStorePassword", (config, prop) -> { + config.setRestKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlRestKeyStoreType", (config, prop) -> { + logger.info("REST key store type: " + prop); + config.setRestKeyStoreType(prop); + }); + propertyConsumerMap.put("mlRestKeyStoreAlgorithm", (config, prop) -> { + logger.info("REST key store algorithm: " + prop); + config.setRestKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlRestTrustStorePath", (config, prop) -> { + logger.info("REST trust store path: " + prop); + config.setRestTrustStorePath(prop); + }); + propertyConsumerMap.put("mlRestTrustStorePassword", (config, prop) -> { + config.setRestTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlRestTrustStoreType", (config, prop) -> { + logger.info("REST trust store type: " + prop); + config.setRestTrustStoreType(prop); + }); + propertyConsumerMap.put("mlRestTrustStoreAlgorithm", (config, prop) -> { + logger.info("REST trust store algorithm: " + prop); + config.setRestTrustStoreAlgorithm(prop); + }); + /** * mlUsername and mlPassword are the default username/password for connecting to the app's REST server (if one @@ -458,7 +558,7 @@ public void initialize() { */ propertyConsumerMap.put("mlTestRestPort", (config, prop) -> { logger.info("Test REST port: " + prop); - config.setTestRestPort(Integer.parseInt(prop)); + config.setTestRestPort(propertyToInteger("mlTestRestPort", prop)); }); propertyConsumerMap.put("mlTestRestServerName", (config, prop) -> { @@ -504,7 +604,7 @@ public void initialize() { propertyConsumerMap.put("mlContentForestsPerHost", (config, prop) -> { logger.info("Content forests per host: " + prop); - config.setContentForestsPerHost(Integer.parseInt(prop)); + config.setContentForestsPerHost(propertyToInteger("mlContentForestsPerHost", prop)); }); propertyConsumerMap.put("mlCreateForests", (config, prop) -> { @@ -519,7 +619,7 @@ public void initialize() { logger.info("Forests per host: " + prop); String[] tokens = prop.split(","); for (int i = 0; i < tokens.length; i += 2) { - config.getForestCounts().put(tokens[i], Integer.parseInt(tokens[i + 1])); + config.getForestCounts().put(tokens[i], propertyToInteger("mlForestsPerHost", tokens[i + 1])); } }); @@ -532,7 +632,7 @@ public void initialize() { String[] tokens = prop.split(","); Map map = new HashMap<>(); for (int i = 0; i < tokens.length; i += 2) { - map.put(tokens[i], Integer.parseInt(tokens[i + 1])); + map.put(tokens[i], propertyToInteger("mlDatabaseNamesAndReplicaCounts", tokens[i + 1])); } config.setDatabaseNamesAndReplicaCounts(map); }); @@ -783,12 +883,12 @@ public void initialize() { propertyConsumerMap.put("mlModulesLoaderThreadCount", (config, prop) -> { logger.info("Modules loader thread count: " + prop); - config.setModulesLoaderThreadCount(Integer.parseInt(prop)); + config.setModulesLoaderThreadCount(propertyToInteger("mlModulesLoaderThreadCount", prop)); }); propertyConsumerMap.put("mlModulesLoaderBatchSize", (config, prop) -> { logger.info("Modules loader batch size: " + prop); - config.setModulesLoaderBatchSize(Integer.parseInt(prop)); + config.setModulesLoaderBatchSize(propertyToInteger("mlModulesLoaderBatchSize", prop)); }); propertyConsumerMap.put("mlCascadeCollections", (config, prop) -> { @@ -899,7 +999,7 @@ public void initialize() { protected void registerDataLoadingProperties() { propertyConsumerMap.put("mlDataBatchSize", (config, prop) -> { logger.info("Batch size for loading data: " + prop); - config.getDataConfig().setBatchSize(Integer.parseInt(prop)); + config.getDataConfig().setBatchSize(propertyToInteger("mlDataBatchSize", prop)); }); propertyConsumerMap.put("mlDataCollections", (config, prop) -> { diff --git a/src/main/java/com/marklogic/appdeployer/command/CommandMapBuilder.java b/src/main/java/com/marklogic/appdeployer/command/CommandMapBuilder.java index fc83b62e..bec875f2 100644 --- a/src/main/java/com/marklogic/appdeployer/command/CommandMapBuilder.java +++ b/src/main/java/com/marklogic/appdeployer/command/CommandMapBuilder.java @@ -161,6 +161,7 @@ private void addCommandsThatDoNotWriteToDatabases(Map> map securityCommands.add(new InsertCertificateHostsTemplateCommand()); securityCommands.add(new DeployExternalSecurityCommand()); securityCommands.add(new DeploySecureCredentialsCommand()); + securityCommands.add(new DeployCredentialsCommand()); securityCommands.add(new DeployPrivilegesCommand()); securityCommands.add(new DeployPrivilegeRolesCommand()); securityCommands.add(new DeployProtectedCollectionsCommand()); diff --git a/src/main/java/com/marklogic/appdeployer/command/SortOrderConstants.java b/src/main/java/com/marklogic/appdeployer/command/SortOrderConstants.java index cf9734a4..61cab12f 100644 --- a/src/main/java/com/marklogic/appdeployer/command/SortOrderConstants.java +++ b/src/main/java/com/marklogic/appdeployer/command/SortOrderConstants.java @@ -33,6 +33,7 @@ public abstract class SortOrderConstants { public static Integer DEPLOY_EXTERNAL_SECURITY = 35; public static Integer DEPLOY_SECURE_CREDENTIALS = 36; + public static Integer DEPLOY_CREDENTIALS = 37; public static Integer DEPLOY_PROTECTED_COLLECTIONS = 40; public static Integer DEPLOY_MIMETYPES = 45; diff --git a/src/main/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsCommand.java b/src/main/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsCommand.java index 6c01948a..c29ecef0 100644 --- a/src/main/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsCommand.java +++ b/src/main/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsCommand.java @@ -49,28 +49,19 @@ public DeployCustomForestsCommand() { @Override public void execute(CommandContext context) { - Configuration configuration = null; - if (context.getAppConfig().getCmaConfig().isDeployForests()) { - configuration = new Configuration(); - } - for (ConfigDir configDir : context.getAppConfig().getConfigDirs()) { File dir = new File(configDir.getBaseDir(), customForestsPath); if (dir != null && dir.exists()) { payloadParser = new PayloadParser(); for (File f : dir.listFiles()) { if (f.isDirectory()) { - processDirectory(f, context, configuration); + processDirectory(f, context); } } } else { logResourceDirectoryNotFound(dir); } } - - if (configuration != null) { - deployConfiguration(context, configuration); - } } /** @@ -79,7 +70,7 @@ public void execute(CommandContext context) { * @param dir * @param context */ - protected void processDirectory(File dir, CommandContext context, Configuration configuration) { + protected void processDirectory(File dir, CommandContext context) { if (logger.isInfoEnabled()) { logger.info("Processing custom forest files in directory: " + dir.getAbsolutePath()); } @@ -91,6 +82,11 @@ protected void processDirectory(File dir, CommandContext context, Configuration } String payload = readResourceFromFile(context, f); + // As of 4.6.1, create a CMA request per file so that the user has control over how many forests are + // submitted in a single request, thus avoiding potential timeouts. + Configuration configuration = context.getAppConfig().getCmaConfig().isDeployForests() ? + new Configuration() : null; + if (payloadParser.isJsonPayload(payload)) { if (configuration != null) { addForestsToCmaConfiguration(context, payload, configuration); @@ -104,6 +100,10 @@ protected void processDirectory(File dir, CommandContext context, Configuration mgr.save(payload); } } + + if (configuration != null) { + deployConfiguration(context, configuration); + } } } diff --git a/src/main/java/com/marklogic/appdeployer/command/security/DeployCredentialsCommand.java b/src/main/java/com/marklogic/appdeployer/command/security/DeployCredentialsCommand.java new file mode 100644 index 00000000..a1888e03 --- /dev/null +++ b/src/main/java/com/marklogic/appdeployer/command/security/DeployCredentialsCommand.java @@ -0,0 +1,43 @@ +/* + * Copyright (c) 2023 MarkLogic Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.marklogic.appdeployer.command.security; + +import com.marklogic.appdeployer.command.AbstractResourceCommand; +import com.marklogic.appdeployer.command.CommandContext; +import com.marklogic.appdeployer.command.SortOrderConstants; +import com.marklogic.appdeployer.command.UndoableCommand; +import com.marklogic.mgmt.resource.ResourceManager; +import com.marklogic.mgmt.resource.security.CredentialsManager; + +import java.io.File; + +public class DeployCredentialsCommand extends AbstractResourceCommand implements UndoableCommand { + + public DeployCredentialsCommand() { + setExecuteSortOrder(SortOrderConstants.DEPLOY_CREDENTIALS); + setUndoSortOrder(SortOrderConstants.DEPLOY_CREDENTIALS); + } + + @Override + protected File[] getResourceDirs(CommandContext context) { + return findResourceDirs(context, configDir -> configDir.getCredentialsDir()); + } + + @Override + protected ResourceManager getResourceManager(CommandContext context) { + return new CredentialsManager(context.getManageClient()); + } +} diff --git a/src/main/java/com/marklogic/mgmt/DefaultManageConfigFactory.java b/src/main/java/com/marklogic/mgmt/DefaultManageConfigFactory.java index 375844d7..33d9c17b 100644 --- a/src/main/java/com/marklogic/mgmt/DefaultManageConfigFactory.java +++ b/src/main/java/com/marklogic/mgmt/DefaultManageConfigFactory.java @@ -56,7 +56,7 @@ public void initialize() { propertyConsumerMap.put("mlManagePort", (config, prop) -> { logger.info("Manage port: " + prop); - config.setPort(Integer.parseInt(prop)); + config.setPort(propertyToInteger("mlManagePort", prop)); }); propertyConsumerMap.put("mlManageAuthentication", (config, prop) -> { @@ -127,19 +127,16 @@ public void initialize() { config.setBasePath(managePath); }); - propertyConsumerMap.put("mlManageScheme", (config, prop) -> { - logger.info("Manage scheme: " + prop); - config.setScheme(prop); - }); - propertyConsumerMap.put("mlManageSimpleSsl", (config, prop) -> { logger.info("Use simple SSL for Manage app server: " + prop); config.setConfigureSimpleSsl(Boolean.parseBoolean(prop)); + config.setScheme("https"); }); propertyConsumerMap.put("mlManageSslProtocol", (config, prop) -> { logger.info("Using SSL protocol for Manage app server: " + prop); config.setSslProtocol(prop); + config.setScheme("https"); }); propertyConsumerMap.put("mlManageSslHostnameVerifier", (config, prop) -> { @@ -156,6 +153,7 @@ public void initialize() { propertyConsumerMap.put("mlManageUseDefaultKeystore", (config, prop) -> { logger.info("Using default JVM keystore for SSL for Manage app server: " + prop); config.setUseDefaultKeystore(Boolean.parseBoolean(prop)); + config.setScheme("https"); }); propertyConsumerMap.put("mlManageTrustManagementAlgorithm", (config, prop) -> { @@ -163,6 +161,72 @@ public void initialize() { config.setTrustManagementAlgorithm(prop); }); + propertyConsumerMap.put("mlKeyStorePath", (config, prop) -> { + logger.info("Manage key store path: " + prop); + config.setKeyStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlKeyStorePassword", (config, prop) -> { + config.setKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlKeyStoreType", (config, prop) -> { + logger.info("Manage key store type: " + prop); + config.setKeyStoreType(prop); + }); + propertyConsumerMap.put("mlKeyStoreAlgorithm", (config, prop) -> { + logger.info("Manage key store algorithm: " + prop); + config.setKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlTrustStorePath", (config, prop) -> { + logger.info("Manage trust store path: " + prop); + config.setTrustStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlTrustStorePassword", (config, prop) -> { + config.setTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlTrustStoreType", (config, prop) -> { + logger.info("Manage trust store type: " + prop); + config.setTrustStoreType(prop); + }); + propertyConsumerMap.put("mlTrustStoreAlgorithm", (config, prop) -> { + logger.info("Manage trust store algorithm: " + prop); + config.setTrustStoreAlgorithm(prop); + }); + + propertyConsumerMap.put("mlManageKeyStorePath", (config, prop) -> { + logger.info("Manage key store path: " + prop); + config.setKeyStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlManageKeyStorePassword", (config, prop) -> { + config.setKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlManageKeyStoreType", (config, prop) -> { + logger.info("Manage key store type: " + prop); + config.setKeyStoreType(prop); + }); + propertyConsumerMap.put("mlManageKeyStoreAlgorithm", (config, prop) -> { + logger.info("Manage key store algorithm: " + prop); + config.setKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlManageTrustStorePath", (config, prop) -> { + logger.info("Manage trust store path: " + prop); + config.setTrustStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlManageTrustStorePassword", (config, prop) -> { + config.setTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlManageTrustStoreType", (config, prop) -> { + logger.info("Manage trust store type: " + prop); + config.setTrustStoreType(prop); + }); + propertyConsumerMap.put("mlManageTrustStoreAlgorithm", (config, prop) -> { + logger.info("Manage trust store algorithm: " + prop); + config.setTrustStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlManageCleanJsonPayloads", (config, prop) -> { logger.info("Cleaning Management API JSON payloads: " + prop); config.setCleanJsonPayloads(Boolean.parseBoolean(prop)); @@ -187,6 +251,11 @@ public void initialize() { config.setSecurityPassword(prop); }); + propertyConsumerMap.put("mlManageScheme", (config, prop) -> { + logger.info("Manage scheme: " + prop); + config.setScheme(prop); + }); + // Processed last so that it can override scheme/port propertyConsumerMap.put("mlCloudApiKey", (config, prop) -> { logger.info("Setting Manage cloud API key and forcing scheme to HTTPS and port to 443"); diff --git a/src/main/java/com/marklogic/mgmt/ManageClient.java b/src/main/java/com/marklogic/mgmt/ManageClient.java index d6081745..3286f1d5 100644 --- a/src/main/java/com/marklogic/mgmt/ManageClient.java +++ b/src/main/java/com/marklogic/mgmt/ManageClient.java @@ -22,6 +22,7 @@ import com.marklogic.rest.util.RestConfig; import com.marklogic.rest.util.RestTemplateUtil; import org.jdom2.Namespace; +import org.springframework.core.io.Resource; import org.springframework.http.*; import org.springframework.util.LinkedMultiValueMap; import org.springframework.util.MultiValueMap; @@ -204,16 +205,28 @@ public String getJsonAsSecurityUser(String path) { .getBody(); } - public void delete(String path) { + public void delete(String path, String... headerNamesAndValues) { logRequest(path, "", "DELETE"); - getRestTemplate().delete(buildUri(path)); + delete(getRestTemplate(), path, headerNamesAndValues); } - public void deleteAsSecurityUser(String path) { + public void deleteAsSecurityUser(String path, String... headerNamesAndValues) { logSecurityUserRequest(path, "", "DELETE"); - getSecurityUserRestTemplate().delete(buildUri(path)); + delete(getSecurityUserRestTemplate(), path, headerNamesAndValues); } + private void delete(RestTemplate restTemplate, String path, String... headerNamesAndValues) { + URI uri = buildUri(path); + HttpHeaders headers = new HttpHeaders(); + if (headerNamesAndValues != null) { + for (int i = 0; i < headerNamesAndValues.length; i += 2) { + headers.add(headerNamesAndValues[i], headerNamesAndValues[i + 1]); + } + } + HttpEntity entity = new HttpEntity<>(null, headers); + restTemplate.exchange(uri, HttpMethod.DELETE, entity, String.class); + } + /** * Per #187 and version 3.1.0, when an HttpEntity is constructed with a JSON payload, this method will check to see * if it should "clean" the JSON via the Jackson library, which is primarily intended for removing comments from diff --git a/src/main/java/com/marklogic/mgmt/admin/AdminManager.java b/src/main/java/com/marklogic/mgmt/admin/AdminManager.java index 570f4ad2..9d2c660b 100644 --- a/src/main/java/com/marklogic/mgmt/admin/AdminManager.java +++ b/src/main/java/com/marklogic/mgmt/admin/AdminManager.java @@ -19,6 +19,7 @@ import com.marklogic.rest.util.Fragment; import com.marklogic.rest.util.RestConfig; import com.marklogic.rest.util.RestTemplateUtil; +import org.apache.commons.lang3.tuple.ImmutablePair; import org.springframework.core.io.ByteArrayResource; import org.springframework.core.io.Resource; import org.springframework.http.*; @@ -80,8 +81,9 @@ public boolean execute() { HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_JSON); HttpEntity entity = new HttpEntity<>(payload, headers); + ImmutablePair credentialsStatus = checkCredentialsAndReplaceNulls(adminConfig); try { - ResponseEntity response = getRestTemplate().exchange(uri, HttpMethod.POST, entity, String.class); + ResponseEntity response = getRestTemplate().exchange(uri, HttpMethod.POST, entity, String.class); logger.info("Initialization response: " + response); // According to http://docs.marklogic.com/REST/POST/admin/v1/init, a 202 is sent back in the event a // restart is needed. A 400 or 401 will be thrown as an error by RestTemplate. @@ -98,12 +100,14 @@ public boolean execute() { logger.error("Caught error, response body: " + body); throw hcee; } - } + } finally { + restoreCredentials(adminConfig, credentialsStatus); + } } }); } - public void installAdmin() { + public void installAdmin() { installAdmin(null, null); } @@ -130,8 +134,9 @@ public boolean execute() { HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_JSON); HttpEntity entity = new HttpEntity<>(payload, headers); + ImmutablePair credentialsStatus = checkCredentialsAndReplaceNulls(adminConfig); try { - ResponseEntity response = getRestTemplate().exchange(uri, HttpMethod.POST, entity, String.class); + ResponseEntity response = getRestTemplate().exchange(uri, HttpMethod.POST, entity, String.class); logger.info("Admin installation response: " + response); // According to http://docs.marklogic.com/REST/POST/admin/v1/init, a 202 is sent back in the event a // restart is needed. A 400 or 401 will be thrown as an error by RestTemplate. @@ -143,6 +148,8 @@ public boolean execute() { return false; } throw hcee; + } finally { + restoreCredentials(adminConfig, credentialsStatus); } } }); @@ -323,4 +330,31 @@ public RestTemplate getRestTemplate() { } return this.restTemplate; } + + private ImmutablePair checkCredentialsAndReplaceNulls(AdminConfig adminConfig) { + boolean setNullUsername = false; + boolean setNullPassword = false; + if (adminConfig.getUsername() == null) { + adminConfig.setUsername(""); + setNullUsername = true; + this.restTemplate = null; + } + if (adminConfig.getPassword() == null) { + adminConfig.setPassword(""); + setNullPassword = true; + this.restTemplate = null; + } + return new ImmutablePair<>(setNullUsername, setNullPassword); + } + + private void restoreCredentials(AdminConfig adminConfig, ImmutablePair credentialsStatus) { + if (credentialsStatus.getLeft()) { + adminConfig.setUsername(null); + this.restTemplate = null; + } + if (credentialsStatus.getRight()) { + adminConfig.setPassword(null); + this.restTemplate = null; + } + } } diff --git a/src/main/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactory.java b/src/main/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactory.java index ece94b97..499d9540 100644 --- a/src/main/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactory.java +++ b/src/main/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactory.java @@ -56,7 +56,7 @@ public void initialize() { propertyConsumerMap.put("mlAdminPort", (config, prop) -> { logger.info("Admin interface port: " + prop); - config.setPort(Integer.parseInt(prop)); + config.setPort(propertyToInteger("mlAdminPort", prop)); }); propertyConsumerMap.put("mlAdminAuthentication", (config, prop) -> { @@ -124,19 +124,16 @@ public void initialize() { config.setBasePath(adminPath); }); - propertyConsumerMap.put("mlAdminScheme", (config, prop) -> { - logger.info("Admin interface scheme: " + prop); - config.setScheme(prop); - }); - propertyConsumerMap.put("mlAdminSimpleSsl", (config, prop) -> { logger.info("Use simple SSL for Admin interface: " + prop); config.setConfigureSimpleSsl(Boolean.parseBoolean(prop)); + config.setScheme("https"); }); propertyConsumerMap.put("mlAdminSslProtocol", (config, prop) -> { logger.info("Using SSL protocol for Admin app server: " + prop); config.setSslProtocol(prop); + config.setScheme("https"); }); propertyConsumerMap.put("mlAdminSslHostnameVerifier", (config, prop) -> { @@ -153,6 +150,7 @@ public void initialize() { propertyConsumerMap.put("mlAdminUseDefaultKeystore", (config, prop) -> { logger.info("Using default JVM keystore for SSL for Admin app server: " + prop); config.setUseDefaultKeystore(Boolean.parseBoolean(prop)); + config.setScheme("https"); }); propertyConsumerMap.put("mlAdminTrustManagementAlgorithm", (config, prop) -> { @@ -160,6 +158,77 @@ public void initialize() { config.setTrustManagementAlgorithm(prop); }); + propertyConsumerMap.put("mlKeyStorePath", (config, prop) -> { + logger.info("Admin key store path: " + prop); + config.setKeyStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlKeyStorePassword", (config, prop) -> { + config.setKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlKeyStoreType", (config, prop) -> { + logger.info("Admin key store type: " + prop); + config.setKeyStoreType(prop); + }); + propertyConsumerMap.put("mlKeyStoreAlgorithm", (config, prop) -> { + logger.info("Admin key store algorithm: " + prop); + config.setKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlTrustStorePath", (config, prop) -> { + logger.info("Admin trust store path: " + prop); + config.setTrustStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlTrustStorePassword", (config, prop) -> { + config.setTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlTrustStoreType", (config, prop) -> { + logger.info("Admin trust store type: " + prop); + config.setTrustStoreType(prop); + }); + propertyConsumerMap.put("mlTrustStoreAlgorithm", (config, prop) -> { + logger.info("Admin trust store algorithm: " + prop); + config.setTrustStoreAlgorithm(prop); + }); + + propertyConsumerMap.put("mlAdminKeyStorePath", (config, prop) -> { + logger.info("Admin key store path: " + prop); + config.setKeyStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlAdminKeyStorePassword", (config, prop) -> { + config.setKeyStorePassword(prop); + }); + propertyConsumerMap.put("mlAdminKeyStoreType", (config, prop) -> { + logger.info("Admin key store type: " + prop); + config.setKeyStoreType(prop); + }); + propertyConsumerMap.put("mlAdminKeyStoreAlgorithm", (config, prop) -> { + logger.info("Admin key store algorithm: " + prop); + config.setKeyStoreAlgorithm(prop); + }); + propertyConsumerMap.put("mlAdminTrustStorePath", (config, prop) -> { + logger.info("Admin trust store path: " + prop); + config.setTrustStorePath(prop); + config.setScheme("https"); + }); + propertyConsumerMap.put("mlAdminTrustStorePassword", (config, prop) -> { + config.setTrustStorePassword(prop); + }); + propertyConsumerMap.put("mlAdminTrustStoreType", (config, prop) -> { + logger.info("Admin trust store type: " + prop); + config.setTrustStoreType(prop); + }); + propertyConsumerMap.put("mlAdminTrustStoreAlgorithm", (config, prop) -> { + logger.info("Admin trust store algorithm: " + prop); + config.setTrustStoreAlgorithm(prop); + }); + + propertyConsumerMap.put("mlAdminScheme", (config, prop) -> { + logger.info("Admin scheme: " + prop); + config.setScheme(prop); + }); + // Processed last so that it can override scheme/port propertyConsumerMap.put("mlCloudApiKey", (config, prop) -> { logger.info("Setting Admin cloud API key and forcing scheme to HTTPS and port to 443"); diff --git a/src/main/java/com/marklogic/mgmt/api/database/DatabaseSorter.java b/src/main/java/com/marklogic/mgmt/api/database/DatabaseSorter.java index 0d76c79d..9876ba56 100644 --- a/src/main/java/com/marklogic/mgmt/api/database/DatabaseSorter.java +++ b/src/main/java/com/marklogic/mgmt/api/database/DatabaseSorter.java @@ -43,6 +43,13 @@ public String[] sortDatabasesAndReturnNames(List databases) { } } - return sorter.sort(); + try { + return sorter.sort(); + } catch (IllegalStateException ex) { + throw new IllegalArgumentException("Unable to deploy databases due to circular dependencies " + + "between two or more databases; please remove these circular dependencies in order to deploy" + + " your databases. An example of a circular dependency is database A depending on database B as its " + + "triggers databases, while database B depends on database A as its schemas database."); + } } } diff --git a/src/main/java/com/marklogic/mgmt/resource/AbstractResourceManager.java b/src/main/java/com/marklogic/mgmt/resource/AbstractResourceManager.java index 0a7d18bc..d8b186c6 100644 --- a/src/main/java/com/marklogic/mgmt/resource/AbstractResourceManager.java +++ b/src/main/java/com/marklogic/mgmt/resource/AbstractResourceManager.java @@ -32,11 +32,17 @@ public abstract class AbstractResourceManager extends AbstractManager implements private ManageClient manageClient; private boolean updateAllowed = true; + protected final boolean usePutForCreate; public AbstractResourceManager(ManageClient client) { - this.manageClient = client; + this(client, false); } + public AbstractResourceManager(ManageClient client, boolean usePutForCreate) { + this.manageClient = client; + this.usePutForCreate = usePutForCreate; + } + public String getResourcesPath() { return format("/manage/v2/%ss", getResourceName()); } @@ -119,7 +125,9 @@ protected SaveReceipt createNewResource(String payload, String resourceId) { logger.info(format("Creating %s: %s", label, resourceId)); } String path = getCreateResourcePath(payload); - ResponseEntity response = postPayload(manageClient, path, payload); + ResponseEntity response = this.usePutForCreate ? + putPayload(manageClient, path, payload) : + postPayload(manageClient, path, payload); if (logger.isInfoEnabled()) { logger.info(format("Created %s: %s", label, resourceId)); } @@ -194,14 +202,15 @@ protected void beforeDelete(String resourceId, String path, String... resourceUr * Convenience method for performing a delete once the correct path for the resource has been constructed. * * @param path + * @param headerNamesAndValues optional sequence of header names and values to be included in the DELETE request. */ - public void deleteAtPath(String path) { + public void deleteAtPath(String path, String... headerNamesAndValues) { String label = getResourceName(); logger.info(format("Deleting %s at path %s", label, path)); if (useSecurityUser()) { - manageClient.deleteAsSecurityUser(path); + manageClient.deleteAsSecurityUser(path, headerNamesAndValues); } else { - manageClient.delete(path); + manageClient.delete(path, headerNamesAndValues); } logger.info(format("Deleted %s at path %s", label, path)); } diff --git a/src/main/java/com/marklogic/mgmt/resource/security/CredentialsManager.java b/src/main/java/com/marklogic/mgmt/resource/security/CredentialsManager.java new file mode 100644 index 00000000..78ca1054 --- /dev/null +++ b/src/main/java/com/marklogic/mgmt/resource/security/CredentialsManager.java @@ -0,0 +1,60 @@ +/* + * Copyright (c) 2023 MarkLogic Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.marklogic.mgmt.resource.security; + +import com.marklogic.mgmt.DeleteReceipt; +import com.marklogic.mgmt.ManageClient; +import com.marklogic.mgmt.resource.AbstractResourceManager; + +public class CredentialsManager extends AbstractResourceManager { + + public CredentialsManager(ManageClient client) { + super(client, true); + } + + @Override + public String getResourcesPath() { + return "/manage/v2/credentials/properties"; + } + + + @Override + protected String getIdFieldName() { + return "type"; + } + + @Override + protected String getResourceId(String payload) { + return getCredentialsType(payload); + } + + @Override + public DeleteReceipt delete(String payload, String... resourceUrlParams) { + final String type = getCredentialsType(payload); + final String path = "/manage/v2/credentials/properties?type=" + type; + // The DELETE endpoint - https://docs.marklogic.com/REST/DELETE/manage/v2/credentials/properties - seems to + // erroneously require a Content-type header, even though there's no request body. + super.deleteAtPath(path, "Content-type", "application/json"); + return new DeleteReceipt(type, path, true); + } + + private String getCredentialsType(String payload) { + if (payloadParser.isJsonPayload(payload)) { + return payloadParser.getPayloadFieldValue(payload, getIdFieldName()); + } + return payloadParser.getPayloadFieldValue(payload, "azure", false) != null ? "azure" : "aws"; + } +} diff --git a/src/main/java/com/marklogic/mgmt/util/PropertySourceFactory.java b/src/main/java/com/marklogic/mgmt/util/PropertySourceFactory.java index c14151e0..99ec0a58 100644 --- a/src/main/java/com/marklogic/mgmt/util/PropertySourceFactory.java +++ b/src/main/java/com/marklogic/mgmt/util/PropertySourceFactory.java @@ -73,4 +73,13 @@ public void setCheckWithMarklogicPrefix(boolean applyMarklogicPrefix) { public PropertySource getPropertySource() { return propertySource; } + + protected final Integer propertyToInteger(String propertyName, String propertyValue) { + try { + return Integer.parseInt(propertyValue); + } catch (NumberFormatException ex) { + throw new IllegalArgumentException(format("The property %s requires a numeric value; invalid value: ‘%s'", propertyName, propertyValue)); + } + } + } diff --git a/src/main/java/com/marklogic/rest/util/Fragment.java b/src/main/java/com/marklogic/rest/util/Fragment.java index c51a260d..2815d106 100644 --- a/src/main/java/com/marklogic/rest/util/Fragment.java +++ b/src/main/java/com/marklogic/rest/util/Fragment.java @@ -63,6 +63,7 @@ public Fragment(String xml, Namespace... namespaces) { list.add(Namespace.getNamespace("sec", "http://marklogic.com/xdmp/security")); list.add(Namespace.getNamespace("ts", "http://marklogic.com/manage/task-server")); list.add(Namespace.getNamespace("t", "http://marklogic.com/manage/tasks")); + list.add(Namespace.getNamespace("creds", "http://marklogic.com/manage/credentials/properties")); list.addAll(Arrays.asList(namespaces)); this.namespaces = list.toArray(new Namespace[] {}); } catch (Exception e) { diff --git a/src/main/java/com/marklogic/rest/util/RestConfig.java b/src/main/java/com/marklogic/rest/util/RestConfig.java index 4152e40e..796d28c7 100644 --- a/src/main/java/com/marklogic/rest/util/RestConfig.java +++ b/src/main/java/com/marklogic/rest/util/RestConfig.java @@ -17,6 +17,7 @@ import com.marklogic.client.DatabaseClientBuilder; import com.marklogic.client.DatabaseClientFactory; +import com.marklogic.client.DatabaseClientFactory.SSLHostnameVerifier; import com.marklogic.client.ext.modulesloader.ssl.SimpleX509TrustManager; import com.marklogic.client.ext.ssl.SslConfig; import com.marklogic.client.ext.ssl.SslUtil; @@ -53,6 +54,16 @@ public class RestConfig { @Deprecated private X509HostnameVerifier hostnameVerifier; + // Added in 4.7.0 for 2-way SSL. + private String keyStorePath; + private String keyStorePassword; + private String keyStoreType; + private String keyStoreAlgorithm; + private String trustStorePath; + private String trustStorePassword; + private String trustStoreType; + private String trustStoreAlgorithm; + public RestConfig() { } @@ -72,6 +83,12 @@ public RestConfig(RestConfig other) { this.cloudApiKey = other.cloudApiKey; this.basePath = other.basePath; + this.authType = other.authType; + this.certFile = other.certFile; + this.certPassword = other.certPassword; + this.externalName = other.externalName; + this.samlToken = other.samlToken; + this.configureSimpleSsl = other.configureSimpleSsl; this.useDefaultKeystore = other.useDefaultKeystore; this.sslProtocol = other.sslProtocol; @@ -79,6 +96,15 @@ public RestConfig(RestConfig other) { this.sslContext = other.sslContext; this.hostnameVerifier = other.hostnameVerifier; this.sslHostnameVerifier = other.sslHostnameVerifier; + + this.keyStorePath = other.keyStorePath; + this.keyStorePassword = other.keyStorePassword; + this.keyStoreAlgorithm = other.keyStoreAlgorithm; + this.keyStoreType = other.keyStoreType; + this.trustStorePath = other.trustStorePath; + this.trustStorePassword = other.trustStorePassword; + this.trustStoreType = other.trustStoreType; + this.trustStoreAlgorithm = other.trustStoreAlgorithm; } public DatabaseClientBuilder newDatabaseClientBuilder() { @@ -94,7 +120,18 @@ public DatabaseClientBuilder newDatabaseClientBuilder() { .withCertificatePassword(getCertPassword()) .withKerberosPrincipal(getExternalName()) .withSAMLToken(getSamlToken()) - .withSSLHostnameVerifier(getSslHostnameVerifier()); + .withSSLHostnameVerifier(getSslHostnameVerifier()) + // These 8 were added in 4.7.0. They do not conflict with the SSL config below; if the user is setting + // these, they won't have a reason to provide their own SSLContext nor request that the default keystore + // be used or simple SSL be used. + .withKeyStorePath(getKeyStorePath()) + .withKeyStorePassword(getKeyStorePassword()) + .withKeyStoreType(getKeyStoreType()) + .withKeyStoreAlgorithm(getKeyStoreAlgorithm()) + .withTrustStorePath(getTrustStorePath()) + .withTrustStorePassword(getTrustStorePassword()) + .withTrustStoreType(getTrustStoreType()) + .withTrustStoreAlgorithm(getTrustStoreAlgorithm()); if (getSslContext() != null) { builder.withSSLContext(getSslContext()); @@ -114,7 +151,8 @@ public DatabaseClientBuilder newDatabaseClientBuilder() { .withSSLContext(StringUtils.hasText(sslProtocol) ? SimpleX509TrustManager.newSSLContext(sslProtocol) : SimpleX509TrustManager.newSSLContext()) - .withTrustManager(new SimpleX509TrustManager()); + .withTrustManager(new SimpleX509TrustManager()) + .withSSLHostnameVerifier(SSLHostnameVerifier.ANY); } else { builder.withSSLProtocol(sslProtocol); } @@ -312,4 +350,132 @@ public String getSamlToken() { public void setSamlToken(String samlToken) { this.samlToken = samlToken; } + + /** + * @return + * @since 4.7.0 + */ + public String getKeyStorePath() { + return keyStorePath; + } + + /** + * @param keyStorePath + * @since 4.7.0 + */ + public void setKeyStorePath(String keyStorePath) { + this.keyStorePath = keyStorePath; + } + + /** + * @return + * @since 4.7.0 + */ + public String getKeyStorePassword() { + return keyStorePassword; + } + + /** + * @param keyStorePassword + * @since 4.7.0 + */ + public void setKeyStorePassword(String keyStorePassword) { + this.keyStorePassword = keyStorePassword; + } + + /** + * @return + * @since 4.7.0 + */ + public String getKeyStoreType() { + return keyStoreType; + } + + /** + * @param keyStoreType + * @since 4.7.0 + */ + public void setKeyStoreType(String keyStoreType) { + this.keyStoreType = keyStoreType; + } + + /** + * @return + * @since 4.7.0 + */ + public String getKeyStoreAlgorithm() { + return keyStoreAlgorithm; + } + + /** + * @param keyStoreAlgorithm + * @since 4.7.0 + */ + public void setKeyStoreAlgorithm(String keyStoreAlgorithm) { + this.keyStoreAlgorithm = keyStoreAlgorithm; + } + + /** + * @return + * @since 4.7.0 + */ + public String getTrustStorePath() { + return trustStorePath; + } + + /** + * @param trustStorePath + * @since 4.7.0 + */ + public void setTrustStorePath(String trustStorePath) { + this.trustStorePath = trustStorePath; + } + + /** + * @return + * @since 4.7.0 + */ + public String getTrustStorePassword() { + return trustStorePassword; + } + + /** + * @param trustStorePassword + * @since 4.7.0 + */ + public void setTrustStorePassword(String trustStorePassword) { + this.trustStorePassword = trustStorePassword; + } + + /** + * @return + * @since 4.7.0 + */ + public String getTrustStoreType() { + return trustStoreType; + } + + /** + * @param trustStoreType + * @since 4.7.0 + */ + public void setTrustStoreType(String trustStoreType) { + this.trustStoreType = trustStoreType; + } + + /** + * @return + * @since 4.7.0 + */ + public String getTrustStoreAlgorithm() { + return trustStoreAlgorithm; + } + + /** + * @param trustStoreAlgorithm + * @since 4.7.0 + */ + public void setTrustStoreAlgorithm(String trustStoreAlgorithm) { + this.trustStoreAlgorithm = trustStoreAlgorithm; + } } diff --git a/src/test/java/com/marklogic/appdeployer/DefaultAppConfigFactoryTest.java b/src/test/java/com/marklogic/appdeployer/DefaultAppConfigFactoryTest.java index 46db8757..e9a01ce5 100644 --- a/src/test/java/com/marklogic/appdeployer/DefaultAppConfigFactoryTest.java +++ b/src/test/java/com/marklogic/appdeployer/DefaultAppConfigFactoryTest.java @@ -19,10 +19,10 @@ import com.marklogic.client.DatabaseClientFactory; import com.marklogic.client.ext.SecurityContextType; import com.marklogic.client.ext.modulesloader.impl.PropertiesModuleManager; -import com.marklogic.mgmt.DefaultManageConfigFactory; -import com.marklogic.mgmt.ManageConfig; import com.marklogic.mgmt.util.SimplePropertySource; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; import java.io.File; import java.util.List; @@ -30,6 +30,7 @@ import java.util.Properties; import java.util.Set; +import static java.lang.String.format; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; @@ -851,6 +852,102 @@ void mlCloudBasePathAndMlRestBasePath() { assertEquals("/my/domain/my/test/server", config.getTestRestBasePath()); } + @Test + void keyStore() { + AppConfig config = configure( + "mlRestKeyStorePath", "/rest.jks", + "mlRestKeyStorePassword", "abc", + "mlRestKeyStoreType", "JKS", + "mlRestKeyStoreAlgorithm", "SunX509", + "mlAppServicesKeyStorePath", "/apps.jks", + "mlAppServicesKeyStorePassword", "123", + "mlAppServicesKeyStoreType", "JKS2", + "mlAppServicesKeyStoreAlgorithm", "SunX5092" + ); + + assertEquals("/rest.jks", config.getRestKeyStorePath()); + assertEquals("abc", config.getRestKeyStorePassword()); + assertEquals("JKS", config.getRestKeyStoreType()); + assertEquals("SunX509", config.getRestKeyStoreAlgorithm()); + assertEquals("/apps.jks", config.getAppServicesKeyStorePath()); + assertEquals("123", config.getAppServicesKeyStorePassword()); + assertEquals("JKS2", config.getAppServicesKeyStoreType()); + assertEquals("SunX5092", config.getAppServicesKeyStoreAlgorithm()); + } + + @Test + void trustStore() { + AppConfig config = configure( + "mlRestTrustStorePath", "/rest.jks", + "mlRestTrustStorePassword", "abc", + "mlRestTrustStoreType", "JKS", + "mlRestTrustStoreAlgorithm", "SunX509", + "mlAppServicesTrustStorePath", "/apps.jks", + "mlAppServicesTrustStorePassword", "123", + "mlAppServicesTrustStoreType", "JKS2", + "mlAppServicesTrustStoreAlgorithm", "SunX5092" + ); + + assertEquals("/rest.jks", config.getRestTrustStorePath()); + assertEquals("abc", config.getRestTrustStorePassword()); + assertEquals("JKS", config.getRestTrustStoreType()); + assertEquals("SunX509", config.getRestTrustStoreAlgorithm()); + assertEquals("/apps.jks", config.getAppServicesTrustStorePath()); + assertEquals("123", config.getAppServicesTrustStorePassword()); + assertEquals("JKS2", config.getAppServicesTrustStoreType()); + assertEquals("SunX5092", config.getAppServicesTrustStoreAlgorithm()); + } + + @Test + void globalKeyStoreAndTrustStore() { + AppConfig config = configure( + "mlKeyStorePath", "/key.jks", + "mlKeyStorePassword", "abc", + "mlKeyStoreType", "JKS1", + "mlKeyStoreAlgorithm", "SunX5091", + "mlTrustStorePath", "/trust.jks", + "mlTrustStorePassword", "123", + "mlTrustStoreType", "JKS2", + "mlTrustStoreAlgorithm", "SunX5092" + ); + + assertEquals("/key.jks", config.getRestKeyStorePath()); + assertEquals("abc", config.getRestKeyStorePassword()); + assertEquals("JKS1", config.getRestKeyStoreType()); + assertEquals("SunX5091", config.getRestKeyStoreAlgorithm()); + assertEquals("/key.jks", config.getAppServicesKeyStorePath()); + assertEquals("abc", config.getAppServicesKeyStorePassword()); + assertEquals("JKS1", config.getAppServicesKeyStoreType()); + assertEquals("SunX5091", config.getAppServicesKeyStoreAlgorithm()); + + assertEquals("/trust.jks", config.getRestTrustStorePath()); + assertEquals("123", config.getRestTrustStorePassword()); + assertEquals("JKS2", config.getRestTrustStoreType()); + assertEquals("SunX5092", config.getRestTrustStoreAlgorithm()); + assertEquals("/trust.jks", config.getAppServicesTrustStorePath()); + assertEquals("123", config.getAppServicesTrustStorePassword()); + assertEquals("JKS2", config.getAppServicesTrustStoreType()); + assertEquals("SunX5092", config.getAppServicesTrustStoreAlgorithm()); + } + + @ParameterizedTest + @CsvSource(delimiter = ':', value = { + "mlAppServicesPort:NaN", + "mlRestPort:NaN", + "mlTestRestPort:NaN", + "mlContentForestsPerHost:NaN", + "mlForestsPerHost:1,NaN,3", + "mlDatabaseNamesAndReplicaCounts:1,NaN,3", + "mlModulesLoaderThreadCount:NaN", + "mlDataBatchSize:NaN" + }) + void numericConfigValues(String propertyName, String propertyValue) { + IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, () -> { + configure(propertyName, propertyValue); + }); + assertTrue(exception.getMessage().contains(format("The property %s requires a numeric value; invalid value: ‘NaN'", propertyName))); + } + private AppConfig configure(String... properties) { return new DefaultAppConfigFactory(new SimplePropertySource(properties)).newAppConfig(); } diff --git a/src/test/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsTest.java b/src/test/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsTest.java index 751eb062..49f3e8ff 100644 --- a/src/test/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsTest.java +++ b/src/test/java/com/marklogic/appdeployer/command/forests/DeployCustomForestsTest.java @@ -51,9 +51,12 @@ public void test() { assertTrue(mgr.exists("sample-app-content-custom-3")); } + /** + * Can examine logging to verify that one request is made per forest. + */ @Test - public void deployWithCma() { - appConfig.getCmaConfig().enableAll(); + public void deployWithoutCMA() { + appConfig.getCmaConfig().setDeployForests(false); test(); } } diff --git a/src/test/java/com/marklogic/appdeployer/command/modules/LoadModulesTest.java b/src/test/java/com/marklogic/appdeployer/command/modules/LoadModulesTest.java index 9580faaa..a3d7981f 100644 --- a/src/test/java/com/marklogic/appdeployer/command/modules/LoadModulesTest.java +++ b/src/test/java/com/marklogic/appdeployer/command/modules/LoadModulesTest.java @@ -161,7 +161,7 @@ public void loadModulesWithAssetFileFilterAndTokenReplacement() { @Test public void testServerExists() { appConfig.getFirstConfigDir().setBaseDir(new File(("src/test/resources/sample-app/db-only-config"))); - appConfig.setTestRestPort(8541); + appConfig.setTestRestPort(8003); initializeAppDeployer(new DeployRestApiServersCommand(true), buildLoadModulesCommand()); appDeployer.deploy(appConfig); diff --git a/src/test/java/com/marklogic/appdeployer/command/security/DeployCredentialsTest.java b/src/test/java/com/marklogic/appdeployer/command/security/DeployCredentialsTest.java new file mode 100644 index 00000000..95cdcd64 --- /dev/null +++ b/src/test/java/com/marklogic/appdeployer/command/security/DeployCredentialsTest.java @@ -0,0 +1,56 @@ +/* + * Copyright (c) 2023 MarkLogic Corporation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.marklogic.appdeployer.command.security; + +import com.marklogic.appdeployer.command.AbstractManageResourceTest; +import com.marklogic.appdeployer.command.Command; +import com.marklogic.mgmt.resource.ResourceManager; +import com.marklogic.mgmt.resource.security.CredentialsManager; +import com.marklogic.rest.util.Fragment; + +import static org.junit.jupiter.api.Assertions.*; + +public class DeployCredentialsTest extends AbstractManageResourceTest { + + @Override + protected ResourceManager newResourceManager() { + return new CredentialsManager(manageClient); + } + + @Override + protected Command newCommand() { + return new DeployCredentialsCommand(); + } + + @Override + protected String[] getResourceNames() { + return new String[]{}; + } + + @Override + protected void afterResourcesCreated() { + Fragment f = manageClient.getXml(new CredentialsManager(manageClient).getResourcesPath()+"?format=xml"); + assertEquals("AWS-ACCESS-KEY", f.getElementValue("/creds:credentials-properties/creds:aws/creds:access-key")); + assertEquals("AZURE-STORAGE-ACCOUNT", f.getElementValue("/creds:credentials-properties/creds:azure/creds:storage-account")); + } + + @Override + protected void verifyResourcesWereDeleted(ResourceManager mgr) { + Fragment f = manageClient.getXml(new CredentialsManager(manageClient).getResourcesPath()+"?format=xml"); + assertNull(f.getElementValue("/creds:credentials-properties/creds:aws/creds:access-key")); + assertNull(f.getElementValue("/creds:credentials-properties/creds:azure/creds:storage-account")); + } +} diff --git a/src/test/java/com/marklogic/mgmt/DefaultManageConfigFactoryTest.java b/src/test/java/com/marklogic/mgmt/DefaultManageConfigFactoryTest.java index a51caca7..2276c1e3 100644 --- a/src/test/java/com/marklogic/mgmt/DefaultManageConfigFactoryTest.java +++ b/src/test/java/com/marklogic/mgmt/DefaultManageConfigFactoryTest.java @@ -16,6 +16,7 @@ package com.marklogic.mgmt; import com.marklogic.client.DatabaseClientFactory; +import com.marklogic.client.DatabaseClientFactory.SSLHostnameVerifier; import com.marklogic.mgmt.util.SimplePropertySource; import org.junit.jupiter.api.Test; @@ -98,12 +99,28 @@ public void sslProperties() { "mlManageTrustManagementAlgorithm", "PKIX" ); + assertEquals("https", config.getScheme()); assertTrue(config.isConfigureSimpleSsl()); assertEquals("TLSv1.2", config.getSslProtocol()); assertTrue(config.isUseDefaultKeystore()); assertEquals("PKIX", config.getTrustManagementAlgorithm()); } + @Test + void simpleSsl() { + ManageConfig config = configure( + "mlManageSimpleSsl", "true", + "mlUsername", "admin", + "mlPassword", "admin" + ); + + assertEquals("https", config.getScheme()); + + DatabaseClientFactory.Bean bean = config.newDatabaseClientBuilder().buildBean(); + SSLHostnameVerifier verifier = bean.getSecurityContext().getSSLHostnameVerifier(); + assertEquals(SSLHostnameVerifier.ANY, verifier, "simpleSsl should default to using the ANY hostname verifier"); + } + @Test public void mlHost() { ManageConfig config = configure("mlHost", "host1"); @@ -254,6 +271,70 @@ void mlCloudBasePathWithManageBasePath() { "mlCloudBasePath."); } + @Test + void keyStore() { + ManageConfig config = configure( + "mlManageKeyStorePath", "/my.jks", + "mlManageKeyStorePassword", "abc123", + "mlManageKeyStoreType", "JKS", + "mlManageKeyStoreAlgorithm", "SunX509" + ); + + assertEquals("/my.jks", config.getKeyStorePath()); + assertEquals("abc123", config.getKeyStorePassword()); + assertEquals("JKS", config.getKeyStoreType()); + assertEquals("SunX509", config.getKeyStoreAlgorithm()); + assertEquals("https", config.getScheme()); + } + + @Test + void trustStore() { + ManageConfig config = configure( + "mlManageTrustStorePath", "/my.jks", + "mlManageTrustStorePassword", "abc123", + "mlManageTrustStoreType", "JKS", + "mlManageTrustStoreAlgorithm", "SunX509" + ); + + assertEquals("/my.jks", config.getTrustStorePath()); + assertEquals("abc123", config.getTrustStorePassword()); + assertEquals("JKS", config.getTrustStoreType()); + assertEquals("SunX509", config.getTrustStoreAlgorithm()); + assertEquals("https", config.getScheme()); + } + + @Test + void globalKeyStoreAndTrustStore() { + ManageConfig config = configure( + "mlKeyStorePath", "/key.jks", + "mlKeyStorePassword", "abc", + "mlKeyStoreType", "JKS1", + "mlKeyStoreAlgorithm", "SunX5091", + "mlTrustStorePath", "/trust.jks", + "mlTrustStorePassword", "123", + "mlTrustStoreType", "JKS2", + "mlTrustStoreAlgorithm", "SunX5092" + ); + + assertEquals("/key.jks", config.getKeyStorePath()); + assertEquals("abc", config.getKeyStorePassword()); + assertEquals("JKS1", config.getKeyStoreType()); + assertEquals("SunX5091", config.getKeyStoreAlgorithm()); + assertEquals("/trust.jks", config.getTrustStorePath()); + assertEquals("123", config.getTrustStorePassword()); + assertEquals("JKS2", config.getTrustStoreType()); + assertEquals("SunX5092", config.getTrustStoreAlgorithm()); + assertEquals("https", config.getScheme()); + } + + @Test + void mlManagePort() { + IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, () -> { + configure("mlManagePort", "NaN"); + }); + assertEquals("The property mlManagePort requires a numeric value; invalid value: ‘NaN'", exception.getMessage()); + } + private ManageConfig configure(String... properties) { return new DefaultManageConfigFactory(new SimplePropertySource(properties)).newManageConfig(); } diff --git a/src/test/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactoryTest.java b/src/test/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactoryTest.java index 8e1e7bb8..91c51822 100644 --- a/src/test/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactoryTest.java +++ b/src/test/java/com/marklogic/mgmt/admin/DefaultAdminConfigFactoryTest.java @@ -15,15 +15,16 @@ */ package com.marklogic.mgmt.admin; -import com.marklogic.client.DatabaseClientFactory; -import com.marklogic.mgmt.util.SimplePropertySource; -import org.junit.jupiter.api.Test; - import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; +import org.junit.jupiter.api.Test; + +import com.marklogic.client.DatabaseClientFactory; +import com.marklogic.client.DatabaseClientFactory.SSLHostnameVerifier; +import com.marklogic.mgmt.util.SimplePropertySource; + public class DefaultAdminConfigFactoryTest { @Test @@ -69,12 +70,28 @@ public void sslProperties() { "mlAdminTrustManagementAlgorithm", "PKIX" ); + assertEquals("https", config.getScheme()); assertTrue(config.isConfigureSimpleSsl()); assertEquals("TLSv1.2", config.getSslProtocol()); assertTrue(config.isUseDefaultKeystore()); assertEquals("PKIX", config.getTrustManagementAlgorithm()); } + @Test + void simpleSsl() { + AdminConfig config = configure( + "mlAdminSimpleSsl", "true", + "mlUsername", "admin", + "mlPassword", "admin" + ); + + assertEquals("https", config.getScheme()); + + DatabaseClientFactory.Bean bean = config.newDatabaseClientBuilder().buildBean(); + SSLHostnameVerifier verifier = bean.getSecurityContext().getSSLHostnameVerifier(); + assertEquals(SSLHostnameVerifier.ANY, verifier, "simpleSsl should default to using the ANY hostname verifier"); + } + @Test void cloudApiKeyAndBasePath() { AdminConfig config = configure( @@ -212,7 +229,72 @@ void mlCloudBasePathWithAdminBasePath() { "mlCloudBasePath."); } + @Test + void keyStore() { + AdminConfig config = configure( + "mlAdminKeyStorePath", "/my.jks", + "mlAdminKeyStorePassword", "abc123", + "mlAdminKeyStoreType", "JKS", + "mlAdminKeyStoreAlgorithm", "SunX509" + ); + + assertEquals("/my.jks", config.getKeyStorePath()); + assertEquals("abc123", config.getKeyStorePassword()); + assertEquals("JKS", config.getKeyStoreType()); + assertEquals("SunX509", config.getKeyStoreAlgorithm()); + assertEquals("https", config.getScheme()); + } + + @Test + void trustStore() { + AdminConfig config = configure( + "mlAdminTrustStorePath", "/my.jks", + "mlAdminTrustStorePassword", "abc123", + "mlAdminTrustStoreType", "JKS", + "mlAdminTrustStoreAlgorithm", "SunX509" + ); + + assertEquals("/my.jks", config.getTrustStorePath()); + assertEquals("abc123", config.getTrustStorePassword()); + assertEquals("JKS", config.getTrustStoreType()); + assertEquals("SunX509", config.getTrustStoreAlgorithm()); + assertEquals("https", config.getScheme()); + } + + @Test + void globalKeyStoreAndTrustStore() { + AdminConfig config = configure( + "mlKeyStorePath", "/key.jks", + "mlKeyStorePassword", "abc", + "mlKeyStoreType", "JKS1", + "mlKeyStoreAlgorithm", "SunX5091", + "mlTrustStorePath", "/trust.jks", + "mlTrustStorePassword", "123", + "mlTrustStoreType", "JKS2", + "mlTrustStoreAlgorithm", "SunX5092" + ); + + assertEquals("/key.jks", config.getKeyStorePath()); + assertEquals("abc", config.getKeyStorePassword()); + assertEquals("JKS1", config.getKeyStoreType()); + assertEquals("SunX5091", config.getKeyStoreAlgorithm()); + assertEquals("/trust.jks", config.getTrustStorePath()); + assertEquals("123", config.getTrustStorePassword()); + assertEquals("JKS2", config.getTrustStoreType()); + assertEquals("SunX5092", config.getTrustStoreAlgorithm()); + assertEquals("https", config.getScheme()); + } + private AdminConfig configure(String... properties) { return new DefaultAdminConfigFactory(new SimplePropertySource(properties)).newAdminConfig(); } + + @Test + void mlAdminPort() { + IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, () -> { + configure("mlAdminPort", "NaN"); + }); + assertEquals("The property mlAdminPort requires a numeric value; invalid value: ‘NaN'", exception.getMessage()); + } } + diff --git a/src/test/java/com/marklogic/mgmt/admin/InitializeMarkLogicTest.java b/src/test/java/com/marklogic/mgmt/admin/InitializeMarkLogicTest.java index eee601fc..6018730f 100644 --- a/src/test/java/com/marklogic/mgmt/admin/InitializeMarkLogicTest.java +++ b/src/test/java/com/marklogic/mgmt/admin/InitializeMarkLogicTest.java @@ -18,6 +18,10 @@ import org.junit.jupiter.api.Test; import com.marklogic.mgmt.AbstractMgmtTest; +import org.springframework.web.client.HttpClientErrorException; + +import static org.junit.jupiter.api.Assertions.*; +import static org.junit.jupiter.api.Assertions.assertEquals; public class InitializeMarkLogicTest extends AbstractMgmtTest { @@ -27,7 +31,33 @@ public class InitializeMarkLogicTest extends AbstractMgmtTest { * to ensure no errors are thrown from bad JSON. */ @Test - public void initAgainstAnAlreadyInitializedMarkLogic() { - adminManager.init(); + void initAgainstAnAlreadyInitializedMarkLogic() { + assertDoesNotThrow(() -> adminManager.init()); } + + @Test + void withNullUsername() { + String originalUsername = adminConfig.getUsername(); + try { + adminConfig.setUsername(null); + HttpClientErrorException exception = assertThrows(HttpClientErrorException.class, () -> adminManager.init()); + assertTrue(exception.getMessage().contains("Unauthorized")); + assertEquals(401, exception.getStatusCode().value()); + } finally { + adminConfig.setUsername(originalUsername); + } + } + + @Test + void withNullPassword() { + String originalPassword = adminConfig.getPassword(); + try { + adminConfig.setPassword(null); + HttpClientErrorException exception = assertThrows(HttpClientErrorException.class, () -> adminManager.init()); + assertTrue(exception.getMessage().contains("Unauthorized")); + assertEquals(401, exception.getStatusCode().value()); + } finally { + adminConfig.setPassword(originalPassword); + } + } } diff --git a/src/test/java/com/marklogic/mgmt/admin/InstallAdminTest.java b/src/test/java/com/marklogic/mgmt/admin/InstallAdminTest.java index 2e331d26..2d31e42d 100644 --- a/src/test/java/com/marklogic/mgmt/admin/InstallAdminTest.java +++ b/src/test/java/com/marklogic/mgmt/admin/InstallAdminTest.java @@ -18,6 +18,9 @@ import org.junit.jupiter.api.Test; import com.marklogic.mgmt.AbstractMgmtTest; +import org.springframework.web.client.HttpClientErrorException; + +import static org.junit.jupiter.api.Assertions.*; public class InstallAdminTest extends AbstractMgmtTest { @@ -27,7 +30,33 @@ public class InstallAdminTest extends AbstractMgmtTest { * again. Instead, a message should be logged and ML should not be restarted. */ @Test - public void adminAlreadyInstalled() { - adminManager.installAdmin("admin", "admin"); + void adminAlreadyInstalled() { + assertDoesNotThrow(() -> adminManager.installAdmin("admin", "admin")); } + + @Test + void withNullUsername() { + String originalUsername = adminConfig.getUsername(); + try { + adminConfig.setUsername(null); + HttpClientErrorException exception = assertThrows(HttpClientErrorException.class, () -> adminManager.installAdmin("admin", "admin")); + assertTrue(exception.getMessage().contains("Unauthorized")); + assertEquals(401, exception.getStatusCode().value()); + } finally { + adminConfig.setUsername(originalUsername); + } + } + + @Test + void withNullPassword() { + String originalPassword = adminConfig.getPassword(); + try { + adminConfig.setPassword(null); + HttpClientErrorException exception = assertThrows(HttpClientErrorException.class, () -> adminManager.installAdmin("admin", "admin")); + assertTrue(exception.getMessage().contains("Unauthorized")); + assertEquals(401, exception.getStatusCode().value()); + } finally { + adminConfig.setPassword(originalPassword); + } + } } diff --git a/src/test/java/com/marklogic/mgmt/api/database/SortDatabasesTest.java b/src/test/java/com/marklogic/mgmt/api/database/SortDatabasesTest.java index 2078d0ed..387712ed 100644 --- a/src/test/java/com/marklogic/mgmt/api/database/SortDatabasesTest.java +++ b/src/test/java/com/marklogic/mgmt/api/database/SortDatabasesTest.java @@ -15,18 +15,17 @@ */ package com.marklogic.mgmt.api.database; -import com.marklogic.mgmt.api.database.Database; -import com.marklogic.mgmt.api.database.DatabaseSorter; -import static org.junit.jupiter.api.Assertions.*; import org.junit.jupiter.api.Test; import java.util.Arrays; import java.util.List; -public class SortDatabasesTest { +import static org.junit.jupiter.api.Assertions.*; + +class SortDatabasesTest { @Test - public void test() { + void test() { Database db1 = new Database(null, "db1"); Database db2 = new Database(null, "db2"); Database triggersDb = new Database(null, "triggers-db"); @@ -44,4 +43,21 @@ public void test() { assertEquals("db2", sortedNames[1]); assertEquals("db1", sortedNames[2]); } + + @Test + void circularDependencies() { + Database db1 = new Database(null, "db1"); + Database db2 = new Database(null, "db2"); + db1.setTriggersDatabase(db2.getDatabaseName()); + db2.setSchemaDatabase(db1.getDatabaseName()); + List databases = Arrays.asList(db1, db2); + + DatabaseSorter sorter = new DatabaseSorter(); + IllegalArgumentException ex = assertThrows(IllegalArgumentException.class, + () -> sorter.sortDatabasesAndReturnNames(databases)); + + assertTrue(ex.getMessage().startsWith("Unable to deploy databases due to circular dependencies between " + + "two or more databases; please remove these circular dependencies in order to deploy your databases."), + "Unexpected error message: " + ex.getMessage()); + } } diff --git a/src/test/resources/sample-app/src/main/ml-config/security/credentials/credentials-aws.json b/src/test/resources/sample-app/src/main/ml-config/security/credentials/credentials-aws.json new file mode 100644 index 00000000..3b64c0c5 --- /dev/null +++ b/src/test/resources/sample-app/src/main/ml-config/security/credentials/credentials-aws.json @@ -0,0 +1,5 @@ +{ + "type": "aws", + "access-key": "AWS-ACCESS-KEY", + "secret-key": "SECRET-KEY" +} diff --git a/src/test/resources/sample-app/src/main/ml-config/security/credentials/credentials-azure.xml b/src/test/resources/sample-app/src/main/ml-config/security/credentials/credentials-azure.xml new file mode 100644 index 00000000..2459c8a9 --- /dev/null +++ b/src/test/resources/sample-app/src/main/ml-config/security/credentials/credentials-azure.xml @@ -0,0 +1,6 @@ + + + AZURE-STORAGE-ACCOUNT + STORAGE-KEY + +