Skip to main content
Version: 7.9

Upgrade From Previous Major Version

Introduction​

This guide addresses upgrades from Resolve Actions Pro 6.x to Actions Pro 7.9. For upgrading from one minor Actions Pro 7 release to another (7.x to 7.x), refer to Upgrading to Minor or Patch Version.

You can upgrade to this latest release of Actions Pro from the following releases:

Upgrade from Actions Pro 6.5 or a later version on the 6 branch is not directly supported. To upgrade, go through the following multistep process:

  • First step: Upgrade from 6.5 to 7.0.0.12.
  • Second step: Upgrade from 7.0.0.12 to 7.9.

Contact your Resolve representative if you are running an earlier version than 6.5.

Prerequisites​

Before upgrading to the current version, check the following.

Back Up Your Deployment​

Make a hard backup of the Actions Pro installation directory, Elasticsearch, and database before upgrading. One way to do that is through VM snapshots.

caution

Due to changes in Elasticsearch, rollback is not possible with this upgrade.

Review the Release Notes​

Review the Release Notes before you start the upgrade.

Review the System Requirements​

Ensure that the system requirements are met before you start the upgrade:

note

Actions Pro 6.x installed on CentOS 6 requires OS upgrade to one of the supported OS versions before upgrading to Actions Pro v7.x.

Validate Production Use Cases​

Between releases, Actions Pro makes regular changes to the JAR files that are part of Actions Pro to introduce new functionality and to provide the latest stability and security updates. This may impact your existing use cases. Identify your use cases and validate full functionality before you upgrade your production environment(s).

Verify the following use cases:

  • Automations
  • Dashboards

Check what JAR files have been updated between your Actions Pro version and the version that you are upgrading to:

Uptime Tasks​

Uptime tasks are tasks that you need to perform while your Actions Pro deployment is still up and running, before you stop the Actions Pro services for the upgrade.

Identify All Connected Components​

Version-mismatched components that try to connect to RabbitMQ (usually external RSRemote instances) will have an adverse effect on the upgrade. To identify the current connections before taking remediating action, use any of the approaches described below.

Identify Connections by IP​

Run the following command:

<actions-pro-home>/rabbitmq/linux64/rabbitmq/sbin/rabbitmqctl list_connections
Identify Connections from the UI​

Log in to Action and from the main menu, go to System Administration > List Registration.

Identify Connections Using RSConsole​

After you log in to the RSConsole command-line utility, run these commands:

CONNECT BROADCAST
Info/info
note

In large environments, it is recommended to either increase the console buffer or record the command line while performing this task as the output might be large.

For more information on how to use RSConsole, see System Administration.

Downtime Tasks​

Downtime tasks are tasks that you need to perform after you stop the Actions Pro services for the upgrade.

Where the tasks involve Blueprint.properties configuration changes, remember to run the configuration script after you make the changes to apply them:

<actions-pro-home>/bin/config.sh

CSRF Guard Properties​

In Actions Pro 7.x, CSRF Guard properties are managed through the blueprint properties file, as the csrfguard.properties file is deleted with the upgrade. You must migrate any custom csrfguard.properties properties that you might have to the following rsview.csrfguard.unprotected section in the Blueprint.properties file.

Use the following rules to migrate the properties:

  • New format is rsview.csrfguard.unprotected.<serial number>=<CSRFguard ALIAS>=/<URL PATH>, where:
    • serial number is ascending integer starting with 1 (2, 3, 4, and so on)
    • CSRFguard ALIAS is an arbitrary property name
    • URL PATH is the protected URL
  • Multi-node RSView instances must match the primary node

Example: rsview.csrfguard.unprotected.1=WikiView=/resolve/service/wiki/view/*

Elasticsearch Settings​

Starting with Actions Pro 7.2, the Actions Pro archiving feature started utilizing Elasticsearch Lifecycle Management instead of RSArchive.

If you are upgrading from Actions Pro version 7.1 or earlier, ensure the following Blueprint.properties configuration changes are in place before starting the upgrade:

  • rssearch.shards=1
note

The property might have been set to 1 or higher in your environment even with RSArchive running. Resources will free up as the older indices with more shards are rolled over. To speed up the process, archive as much worksheet data as you can using RSArchive prior to the upgrade.

Operating System Tasks​

Review the list of OS settings below to ensure that your system meets the minimum requirements. If it doesn’t, run the <actions-pro-home>/bin/setup_limits.sh and <actions-pro-home>/bin/setup_sysctl.sh commands as root to set higher limits.

  • FILE LIMIT=131072
  • MEMORY_LIMIT=unlimited
  • ADDRESS_LIMIT=unlimited
  • PROCESS_LIMIT=10240
  • ulimit = 131072
  • vm.max_map_count = 262144

Network Connections​

Ensure the network connectivity between the database and the Actions Pro cluster machines works without issue.

Elasticsearch Data Migration Requirements​

As of Actions Pro 7.0, the internal Elasticsearch (ES) database has been upgraded from 5.x to 7.x. For Actions Pro to function properly when updating from Actions Pro 6.x, existing ES data is migrated automatically to the new ES format.

To minimize the performance impact and to ensure data integrity, the data migration is performed by standing up a new ES cluster and migrating the data from the existing ES cluster to the new one. On each ES node, ensure the following system requirements are met:

  • Memory: double the current rssearch.run.Xmx value and then add 4 GB more temporary RAM on top of it.
  • Disk space: double the capacity used by the old ES version and then add 40% headroom.
    • Example: If the current disk size is 10 GB and 6 GB is allocated to ES, the new disk space should be at least 20 GB.
  • System load: ensure that during the time of migration the system is not under a heavy load.
  • Temporary network ports:
    • Open ports for communication between the existing and the new ES clusters during data migration. You can close the ports after successful migration.
    • By default, the new ports are the current ports +400.
    • The current ports are defined in the blueprint file. For the default values (9200 and 9300), the new ports to open are 9600 and 9700.

Because of the migration, the Actions Pro upgrade consists of two steps:

  1. Data migration: Migrating ES data to a new ES cluster. The existing Actions Pro cluster continues to function during the process.
  2. Upgrading Actions Pro: Upgrading to the new Actions Pro release together with all daily indices and all other non-daily indices data.
caution

Too many ES indices will cause the upgrade to fail. As a rule of thumb, you are likely to have too many indices if you have set the rsarchive.archive.expiry Blueprint property to more than 30 days.

Do the following to reduce the index size before running the upgrade:

  • Run RSArchive to ensure old index data is archived. Do either of the following:
    • Wait for the next scheduled run to happen.
    • Change the schedule to run sooner and then restart RSArchive.
  • If you have set rsarchive.archive.expiry to more than 30 days, consider deleting older indices:
    • For a list of index names and the naming format, see the RSArchive guide for the version that you are upgrading from.
    • Use curl to delete one or more of the indices. Use wildcards to delete multiple files. For example:
      curl -XDELETE http://<ES URL>:9200/worksheet-202108*
    • The Actions Pro ES index data consists of non-daily and daily index data:
      • The daily index is data related to worksheets, process requests, task results, and others. A new index is created for each calendar day and its index name ends in _yyyyMMdd.
      • The non-daily index data is stored in ES without by-day separation. Its index name does not contain a date.

Groovy Sandbox File Backup​

If you are upgrading from Actions Pro 7.6 or an earlier 7.x release, you need to take additional steps in relation to Groovy Sandbox whitelisting improvements made in Actions Pro 7.7.

You need to back up your Groovy Sandbox configuration before starting the upgrade. Back up the following files for each Actions Pro component that has Groovy Sandbox enabled:

  • groovy_blacklist
  • groovy_whitelist
  • groovy_sandbox_properties

These files will be overwritten with defaults for RSArchive, RSConsole, RSControl, RSMgmt, RSRemote, and RSView during the upgrade.

After the upgrade, you can restore your original files or append to the new default files.

Running the Upgrade​

This section contains information about the preparation phase of the migration. It also instructs about upgrading clustered environments and the related procedure.

Upgrading a Standalone Deployment​

To upgrade your standalone Actions Pro deployment, take the following steps:

  1. Ensure that Actions Pro is down (including all standalone RSRemote instances). Run the following on each node:
    Refer to Identify All Connected Components for help. 
    # Stop the services
    <actions-pro-home>/bin/stop.sh all
    # Check if the services have indeed stopped
    <actions-pro-home>/bin/status.sh all
  2. Copy the Actions Pro upgrade ZIP file to your Actions Pro installation directory.
  3. Run the following command from the Actions Pro installation directory: 
    <actions-pro-home>/bin/update.sh -u <rsconsole admin name> -p <rsconsole admin password> -f <upgrade zip file>

The upgrade script accepts the following options:

Option

Description

-u <username>

Where <username> is the username for logging into the RSConsole. 

-p <password>

Use the default password if it hasn't been changed intentionally in a previous upgrade.

note

Ensure the username/password is correct. Otherwise several data migration steps may fail and the user will need to reapply the upgrade.

-f upgrade-x-x-x-x.zip

The name of the upgrade ZIP file.

-dns <hostname>

DNS host name to set into RSView properties for system URL (if not already set).

-port <port>

Host port to set into RSView properties for system URL (if not already set).

-lic

Enter license files (optional).

Continue

An upgrade option that should only be used prior to consulting with Resolve Support.

Upgrading Clustered Deployments​

Items to consider before you start the clustered upgrade:

  • During the upgrade, the primary server will update the Actions Pro database, run data migration, and import the latest base Actions Pro imports. Depending on the server speed, this process might take 30+ minutes. You can monitor the process from the primary server’s rsview.log, rsmgmt.log and update.log.
  • All the secondary Actions Pro servers within the cluster will wait until the primary has finished the upgrade tasks before upgrading themselves.
  • Actions Pro does not support an upgrade on one server while other servers run an older Actions Pro version.

To upgrade your clustered Actions Pro deployment, take the following steps simultaneously on all Actions Pro machines in the cluster:

  1. Ensure that Actions Pro is down (including all standalone RSRemote instances). Run the following on each node:
    Refer to Identify All Connected Components for help.
    # Stop the services
    <actions-pro-home>/bin/stop.sh all
    # Check if the services have indeed stopped
    <actions-pro-home>/bin/status.sh all
  2. Copy the Actions Pro upgrade ZIP file to your Actions Pro installation directory.
  3. Run the following command from the Actions Pro installation directory:
    ```bash
    <actions-pro-home>/bin/update.sh -u <rsconsole admin name> -p <rsconsole admin password> -f <upgrade zip file>
    ```
    At any points of the upgrade process, if a task times out, it may display an option to continue. For example:
WARNING!!!  RSSearch Status is unavailable or 'red'.

If you want to continue with the Update, manually Monitor the Elasticsearch Cluster Status and Hit Y when the Status is at least Yellow.

To Cancel the Update hit N. (Y/N):

At this point, type N to cancel the current upgrade or provide no answer while you research and resolve the issue in a separate session. After you resolve the issue, type Y to continue the upgrade process.

Reapplying an Upgrade​

This Actions Pro version performs data migration and reindexing of Actions Pro data during the upgrade. If the upgrade fails, take the following steps:

  1. Research and determine the issue that prevented the successful upgrade.

  2. After correcting all errors, run the upgrade again with the --continue option to make sure all upgrade tasks are completed.

    bin/update.sh –u <rsconsole user> -p <rsconsole password> -f <update file> --continue

  3. If the upgrade still fails, restore previous Actions Pro version from the backup and reapply the upgrade.

info

Without the --continue option, the upgrade might fail when it encounters updated files it has already applied. This is an example of a typical message:

WARNING!!! Update File <Update File Name> Has Not Been Moved/Backed Up, Cancelling Update

Post-Upgrade Steps​

After the Actions Pro upgrade process is complete, there are several required and optional steps to take to finish the installation.

Required Actions​

MariaDB Driver URL Scheme Update​

The latest version of the MariaDB driver that comes with Actions Pro removes the support for the jdbc:mysql scheme in the JDBC URL, replacing it with jdbc:mariadb instead. Using the old scheme results in an error indicating that it cannot find the driver, due to the mariadb-java-client JAR file removing the MySQLDataSource class.

You need to update the DB URL scheme if you are using MariaDB as the Actions Pro DB and you are upgrading from the following versions:

  • Any version preceding Actions Pro 7.6.0

You don't need to take any action if you are upgrading from Actions Pro 7.6.0 or later.

The upgrade updates most DB URL schemes in the product automatically, including the DB_URL property in blueprint.properties if it was set to jdbc:mysql, but does not update URLs in the following resource categories:

  • Database Gateway settings
  • ActionTasks and other content

See the following sections to learn how to update these resources.

Updating Database Gateway Settings​

You can use an RSConsole script to automatically update any jdbc:mysql schemes in your Database Gateway settings.

  1. Log in to any of the core machines that have the RSConsole component installed.
  2. Start RSConsole and then log in to it:
    <actions-pro-home>/rsconsole/bin/run.sh
  3. Change to the management directory:
    CD mgmt
  4. Run the scheme auto-update command:
    UpdateDBConnPool.groovy
Updating ActionTasks and Other Content​

The potential types of content where you could have used the outdated Database Connection Pool URL in Action Pro are as follows:

  • Wiki Documents
  • ActionTasks
  • System Scripts
  • Task Properties
  • Legacy Gateways
  • SDK2 Push Gateways
  • SDK2 Pull Gateways
  • SDK2 MSG Gateways

You will have to update the Database Connection Pool URL in these locations manually.

To make the process easier, Resolve provides the JDBC MySql Check automation on Resolve Exchange that can find and list all instances of the outdated Database Connection Pool URL in your code.

For example, you might have the following code in an ActionTask content section:

try (Connection conn = DriverManager.getConnection( "jdbc:mysql://127.0.0.1:3306/test", "root", "password")) {
...

To keep using this code after the upgrade, you would need to update it as follows:

try (Connection conn = DriverManager.getConnection( "jdbc:mariadb://127.0.0.1:3306/test", "root", "password")) {
...

RSView Reindexing​

After the initial upgrade is complete, a full reindexing will occur on the primary RSView system. Let the RSView reindexing finish:

  • Do not stop the primary RSView until this indexing is complete.
  • Look for messages indicating reindexing progress in rsview.log.

Browser Cache​

After you have upgraded Actions Pro to the latest version, clear your browser cache before accessing the user interface.

Optional Actions​

  • Linux services:
  • Review the security suggestions in ::title.
  • ​​​Log4j2 guidance
    • Starting with Actions Pro 6.5, Actions Pro moved from Log4j to Log4j v2 for it's core logging framework. This change means that any existing SDK Gateway builds will not be supported anymore. Request the latest gateway builds from Resolve Support and install them using the following steps to ensure Gateways are up and running after upgrade.

The following steps can be used as a guide to upgrading gateway builds to make them compatible with Actions Pro 6.5 and later. Performed the steps for each installation of Actions Pro that has RSRemote and standalone installations of RSRemote.

  1. For an Actions Pro installation, stop RSRemote, RSView, and RSControl running on the server. For a standalone RSRemote installation, just stop RSRemote.
    <actions-pro-home>/bin/stop.sh rsremote rsview rscontrol
  2. Copy the .jar files from /opt/resolve/rsgateways/<gateway-name>/lib/ to /opt/resolve/gatewaylibs, overwriting any existing files.
  3. Open /opt/resolve/rsgateways/<gateway-name>/config/blueprint.properties for editing. This blueprint is provided for reference and is not used by the application. Compare it with the rsremote/resolve blueprint properties. Add any new properties from rsgateways/<gateway-name>/config/blueprint.properties to rsremote/resolve blueprint properties.\
  4. Save the blueprint changes to the main blueprint configuration file.
  5. Run the configuration script:
    <actions-pro-home>/bin/config.sh
  6. Start the components:
    <actions-pro-home>/bin/run.sh rsremote rsview rscontrol

Troubleshooting ​

This section gives instructions related to data migration and Actions Pro upgrade issues and how to resolve them.

General Upgrade Errors​

The following are some of the error messages that may occur during the upgrade and how they can be resolved. Each of these error messages may be preceded by WARNING!!! in the output.

Error MessageResolution
File upgrade-7.x.0.0.zip does not appear to be an Update file.

Actual file name may vary.		Ensure that the upgrade file is in the format: upgrade-x.x.x.x.zip.

Remove /tmp/update.xml from the Action installation directory and then rerun the upgrade.
Current User ID does not match the resolve.user property specified in the blueprint.propertiesCheck that you are logged in as the user specified in resolve.user in the blueprint.properties.
Unable to replace INSTALLDIR in RSMgmt Config file, configuration may failConfiguration might have failed. Check update.log for details.
The RSMQ Instance does not appear to have started properlyValidate that RSMQ is up and running.
Failed to run Index ConfigureRSSearch index may not be configured, Runbook executions and reads might fail. Check rsview.log for details and restart the upgrade after resolving the root issue.
Failed to Re-Index Action Task Data, Result Display May not Behave as ExpectedWorksheet data may not display. Check rsview.log for details and restart the upgrade after resolving the root issue.
Unable to Monitor RSSearch Task Data Re-Index StatusWorksheet data may not display. Check update.log for details and restart the upgrade after resolving the root issue.
Unexpected Error During Update/Upgrade Process. See update.log for more detailsGeneric exception. Check update.log for details and restart upgrade after resolving the root issue.
Execute Update Process FailedContact Resolve Support.
Resolve Update Failed in x secondsContact Resolve Support.
If rsremote stdout log displays the following exception:

java.lang.NoSuchFieldError: log

       at com.resolve.gateway.splunk.ConfigReceiveSplunk.load(ConfigReceiveSplunk.java:214)

       at com.resolve.rsremote.Main.loadSDKDevelopedGateway(Main.java:1003)

       at com.resolve.rsremote.Main.loadSDKDevelopedGateways(Main.java:1065)		Check that only one .jar file is present for the gateway that throws the exception. The above is an example; other gateways may throw the same error.

In the example, Splunk is throwing the exception, so make sure that only one Splunk gateway .jar file is present.

A possible reason for the issue is because some of the jars might have been renamed, in which case the old .jar file is not removed automatically. Ensure that the old .jar file has been removed and only the new .jar file is present in the folder.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/resolve/rsmgmt/lib/groovy-all-2.4.15.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release		Use of a deprecated feature from one of the libraries we use.

This warning can be safely ignored.
SEVERE: Could not contact [localhost:[8005]]. Tomcat may not be running.Jul 07, 2020 2:25:51 AM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.Net.connect0(Native Method)
    at java.base/sun.nio.ch.Net.connect(Net.java:493)
    at java.base/sun.nio.ch.Net.connect(Net.java:482)
    at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:339)
    at java.base/java.net.Socket.connect(Socket.java:603)
    at java.base/java.net.Socket.connect(Socket.java:552)
    at java.base/java.net.Socket.<init>(Socket.java:475)
    at java.base/java.net.Socket.<init>(Socket.java:249)
    at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:491)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
    at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:408)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:497)		Appears when RSView was stopped manually before the upgrade was started.

The error appears because the upgrade script failed to stop RSView which was already stopped.
epmd: Cannot connect to local epmdAppears when RabbitMQ was stopped manually before the upgrade was started.The error appears because the upgrade script failed to stop RabbitMQ which was already stopped.
[WARN ][o.e.c.l.LogConfigurator  ] [<hostname>] Some logging configurations have %marker but don't have %codename. We will automatically add %node name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behaviour in 7.0. You should manually replace %node_name with [%node_name]%marker in these locations:
 /opt/resolve/elasticsearch. New/config/log4j2.properties
Started ElasticSearch pid: 3445		Use of a deprecated feature from one of the libraries we use.

This warning can be safely ignored.
./run.sh: line 30: [: -gt: unary operator expectedUsed for compatibility purposes on some systems.

These errors can be safely ignored.
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.		Use of a deprecated feature from one of the libraries we use.

This warning can be safely ignored.
Cannot Find sree properties file: /opt/resolve/tomcat/webapps/sree/WEB-INF/classes/sree.properties
Cannot Find newsree properties file: /opt/resolve/tomcat/webapps/sree/WEB-INF/classes/newsree.properties
Cannot Find dashboard file: /opt/resolve/bin/dashboard.properties		Used for compatibility purposes with older versions of Actions Pro.

This error can be safely ignored.
Response for creation of daily index worksheet_20200703{"error":{"root_cause":
[{"type":"resource_already_exists_exception",
"reason":"index [worksheet_20200703/q0MCSc-2TAadq2p4FZWt7w] already exists",
"index_uuid":"q0MCSc-2TAadq2p4FZWt7w","index":"worksheet_20200703"}],
"type":"resource_already_exists_exception",
"reason":"index [worksheet_20200703/q0MCSc-2TAadq2p4FZWt7w] already exists",
"index_uuid":"q0MCSc-2TAadq2p4FZWt7w","index":"worksheet_20200703"},"status":400}
Response for alias worksheet_20200703: {"acknowledged":true}
Failed : HTTP error code : 400		The ElasticSearch migration failed to create an index as it already existed.

Errors referring to already-existing resources during the creation of daily indexes can be safely ignored.

Data Migration Errors​

Refer to the table below to repair operational problems during upgrade.

Error MessageResolution
Unexpected Exception Running Update Command - Can't DROP 'u_name'; check that column/key exists

java.sql.SQLSyntaxErrorException: Can't DROP 'u_name'; check that column/key exists at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:138) at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)		This error is acceptable. The upgrade will continue successfully.
2017-10-05 00:36:31,503 DEBUG [main] (Main.java:1011) - onetime 

file exists: true 

2017-10-05 00:36:31,505 ERROR [main] (Main.java:1272) -  Unexpected Exception Running Update Command - You have an error in your SQL  syntax; check the manual that corresponds to your MySQL server version for  the right syntax to use near '(3)' at line 1_Query is:_sql : 'ALTER TABLE  archive_action_result MODIFY COLUMN sys_updated_on DATETIME(3)'

java.sql.SQLSyntaxErrorException: You have an error in your SQL  syntax; check the manual that corresponds to your MySQL server version for  the right syntax to use near '(3)' at line 1 

Query is:

sql : 'ALTER TABLE archive_action_result MODIFY COLUMN
  sys_updated_on DATETIME(3)'at
  org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:138)		The upgrade will continue successfully.
2017-10-12 15:32:07,813 ERROR [main] (Compress.java:365) - Failed to zip directory: java.io.FileNotFoundException:
/opt/Resolve6503/tomcat/webapps/Resolve/mcp (No such file or directory)

2017-10-12 15:32:07,813 DEBUG [main] (?:?) - Removing
/opt/Resolve6503/tomcat/webapps/Resolve/mcp		This error is acceptable.

A back up was attempted, however the mcp javascript doesn't exist in the specified version (6.5.0.3 in the example (Resolve6503).
Waiting Indefinitely for New ES Cluster to have 'green' status……If this message persists, then the new ES (7.x series) has not properly started. The log files under elasticsearch.new/logs/ contain more information.