Installing the Model9 agent
Prerequisites
License key
Obtain your license key from Model9 by opening a “new license request” in the Model9 service portal: https://model9.atlassian.net/servicedesk/customer/portals
The output of the z/OS command “D M=CPU” is required for obtaining a new license.
Java version
Verify that the z/OS Java is version 8 64-bit SR5 FP16 or above. Verify the Java version and change the Java installation directory using the following commands:
cd /usr/lpp/java/J8.0_64/bin ./java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 8.0.5.27 - pmz6480sr5fp27-20190104_01(SR5 FP16)) IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20181219_405297 (JIT enabled, AOT enabled) OpenJ9 - 3f2d574 OMR - 109ba5b IBM - e2996d1) JCL - 20190104_01 based on Oracle jdk8u191-b26
UTC date and time
The object storage protocol requires the z/OS USS and object storage UTC date and time to match. Use the following USS command to display and verify the UTC date and time:
date -u
Note
If the UTC date and time do not match, the agent will fail with an HTTP error code 403 while trying to connect to the object storage.
JZOS setup
The Agent is invoked using the JZOS Batch Java Launcher which is a component of the IBM JVM for z/OS. To verify that the JZOS is correctly configured to run the Agent, check the JZOS JCL procedures and make sure member JVMPRC86 is located in a site standard JES PROCLIB. If any of the JZOS resources are missing, copy them from the Java installation to your installation:
//M9JAVAP JOB ACCT#,SYSPROG,TIME=NOLIMIT,REGION=0M, // NOTIFY=&SYSUID,MSGLEVEL=(1,1),MSGCLASS=X //COPYFILE EXEC PGM=IKJEFT01 //IN DD PATH='/usr/lpp/java/J8.0_64/mvstools/samples/jcl/JVMPRC86' //OUT DD DISP=SHR,DSN=SYS1.PROCLIB(JVMPRC86) //SYSTSPRT DD SYSOUT=* //SYSTSIN DD * OCOPY INDD(IN) OUTDD(OUT) TEXT /* //
Note
For more details about JZOS setup, refer to the JZOS Installation and User Guide: http://publibfp.dhe.ibm.com/epubs/pdf/ajvc0120.pdf
Step 1: Allocate and mount the ZFS filesystem
Create a dedicated ZFS filesystem for the Model9 Agent installation and mount point for the new ZFS, as demonstrated in the following JCL. Edit the JCL using the site’s standard naming convention for the mount point, ZFS and volume serial number if it is not managed by the SMS.
//M9AGTZFS JOB ACCT#,SYSPROG,TIME=NOLIMIT,REGION=0M, // NOTIFY=&SYSUID,MSGLEVEL=(1,1),MSGCLASS=X //* Create a Model9 installation directory //CREATE EXEC PGM=BPXBATCH, // PARM='SH mkdir -p /usr/lpp/model9' //* Define and format a new zFS for Model9 //DEFINE EXEC PGM=IDCAMS //SYSPRINT DD SYSOUT=* //SYSIN DD * DEFINE CLUSTER (NAME(SYS2.MODEL9.ZFS) - VOLUMES(xxxxxx) - LINEAR CYL(200 50)) /* //FORMAT EXEC PGM=IOEAGFMT,REGION=0M, // PARM=('-aggregate SYS2.MODEL9.ZFS -compat') //SYSPRINT DD SYSOUT=* //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //* Mount the newly defined zFS to the Model9 installation directory //MOUNT EXEC PGM=IKJEFT01,DYNAMNBR=10 //SYSTSPRT DD SYSOUT=* //SYSTSIN DD * MOUNT FILESYSTEM('SYS2.MODEL9.ZFS') + MOUNTPOINT('/usr/lpp/model9') + TYPE(ZFS) MODE(RDWR) /* AUTOMOVE */ //
Note
The parameters for the IOEAGFMT utility and the mount commands are case sensitive.
To automatically and permanently mount the ZFS after an IPL, add the mount command to the BPXPRMxx as shown in the following example:
MOUNT FILESYSTEM('SYS2.MODEL9.ZFS') TYPE(ZFS) MODE(RDWR) MOUNTPOINT('/usr/lpp/model9') /* AUTOMOVE */
Note
For a Sysplex environment, remove the comment marks from the term AUTOMOVE in the JCL above to enable file system migration between Sysplex members.
Step 2: Upload the Model9 agent TAR file to the mainframe
Use an FTP utility to upload the Model9 agent’s installation tar file to the model9 directory created in the previous step. Use Passive Mode if supported by the FTP client. The tar file must be uploaded in binary mode as shown in the following example:
$ ftp mf-lp1 Connected to mf-lp1. 220-FTPD1 IBM FTP CS V2R2 at mf-lp1, 06:20:40 on 2017-02-23. 220 Connection will not timeout. Name (mf-lp1:m9user): m9user 331 Send password please. Password: 230 M9U is logged on. Working directory is "M9U.". Remote system type is MVS. ftp> cd /usr/lpp/model9/ 250 HFS directory /usr/lpp/model9/ is the current working directory ftp> bin 200 Representation type is Image ftp> put model9-v1.8.1_build_48c2c57f-agent.tar local: model9-v1.8.1_build_48c2c57f-agent.tar remote: model9-v1.8.1_build_48c2c57f-agent.tar 229 Entering Extended Passive Mode (|||1026|) 125 Storing data set /usr/lpp/model9/model9-v1.8.1_build_48c2c57f-agent.tar 250 Transfer completed successfully. 4483584 bytes sent in 00:02 (1.95 MiB/s) ftp> quit 221 Quit command received. Goodbye.
Step 3: Extract the agent files from the TAR file
Use the tar command in the z/OS UNIX shell to extract the agent files. During extraction, all agent files are saved in the same directory and can only be run by users who have the following permissions in the FACILITY class:
BPX.FILEATTR.APF ACC(READ)
BPX.FILEATTR.PROGCTL ACC(READ)
BPX.FILEATTR.SHARELIB ACC(READ)
The directory name usually includes the agent’s release number. It is recommended to create an alias that does not include the release number for the directory. This makes future upgrades more transparent, as shown in the following example:
TSO OMVS su cd /usr/lpp/model9/ tar -xpf model9-v1.8.1_build_48c2c57f-agent.tar # define the alias name “agent” ln -s model9-v1.8.1_build_48c2c57f-agent agent
Step 4: Create and populate the configuration directory
Create and copy the Model9 sample configuration directory using the following commands:
cd /usr/lpp/model9 mkdir conf cp agent/sampleConf/* conf/
Step 5: Create a program-controlled version of PAX
The Model9 z/OS UNIX files backup uses PAX. This functionality requires creation of a program-controlled version of PAX. Use the following commands:
cd /usr/lpp/model9 mkdir bin cp -p /bin/pax bin # Set program-control flag extattr +p bin/pax
Step 6: Copy the Model9 libraries from USS to PDS
Edit the JCL CPY#PDS located in /usr/lpp/model9/agent/installation/ to create the Model9 LOADLIB, SAMPLIB and EXEC PDS files. Submit the JCL.
Step 7: Install the Model9 Command Line Interface
The Model9 CLI Command Line Interface provides an interface for issuing Model9 commands from TSO / JCL. Install the feature by creating a listener directory under the agent’s main path:
cd /usr/lpp/model9/ mkdir listener chmod 777 listener chmod +t listener
Customize the M9CLI rexx in the EXEC PDS to match installation standards:
fifodir = "/usr/lpp/model9/listener"
loaddir = "SYS2.MODEL9.V181.LOADLIB"
Note
Copy the M9CLI EXEC to your site’s local EXEC library.
Step 8: Define the agent to RACF
Use the sample JCL M9USERST in the SAMPLIB PDS to define the security settings required by the agent. M9USERST permits discrete profiles. If your site uses generic profiles, update the JCL accordingly. If your site uses unprotected resources, no further action is needed.
Note
Review and update the JCL to match local site standards. The SHARED parameter is specified by default. The SHARED.IDS permission is required to run the JCL.
Submit the JCL.
Step 9: Customize the Model9 agent start procedure
Copy the sample JCL M9AGENT from the SAMPLIB PDS to a local PROCLIB member. Update the JCL:
Update | Description |
---|---|
DD STEPLIB | Model9 installation LOADLIB |
PWD environment variable | Model9 agent’s installation path |
CONF_HOME environment variable* | Model9 agent’s configuration directory path |
*CONF_HOME can be used for activating more than one agent in the same LPAR. The parameter will allow the agents to use the same Model9 installation files and libraries, but have a different configuration directory. The recommendation is to have one agent per LPAR, and to have all agents in the same GRS-complex point to the same Model9 complex. However, additional agents in the same LPAR may be required if using a sub-plex, having both development and production environments or pointing the agents to different cloud storage. CONF_HOME must precede the stdenv-main.sh statement. The following is a sample use of the parameter:
//STDENV DD * export PWD=/usr/lpp/model9/agent export CONF_HOME=$PWD/../conf export ENV=agent . $PWD/scripts/stdenv-main.sh //
Note
Add the agent procedure to the system’s startup process.
Step 10: Update the Model9 agent configuration
Before starting the agent, update the following configuration files located in /usr/lpp/model9/conf:
model9-stdenv.sh
Parameter | Description | Required | Default |
---|---|---|---|
JAVA_HOME | 64-bit java home path | Yes | None |
TIME_ZONE | Java time zone, e.g. America/New_York | Yes | None |
TCPIP_NAME | TCP/IP stack used by the agent * | No | TCPIP |
AGENT_MEMORY | Allocated Java heap space for the agent. 1 GB is the minimum value. | No | 1g |
LIFE_CYCLE_MEMORY | Allocated Java heap space for life cycle management. 256 MB is the minimum value. | No | 256m |
SAPI_MEMORY | Allocated Java heap space for the server API (M9SAPI). 56 MB is the minimum value. | No | 56m |
SAPI_NETWORK_DEBUG | Enable network level debug for server API (M9SAPI). Expected values: “true” | “false” | No | false |
SAPI_KEYRING_NAME | The SAF keyring name to be used as a truststore during server API certificate validation. if specified, the keyring will be searched at: &SYSUID/<KeyRingName> | No | None |
* If running in an environment with multiple TCP/IP stacks, use the DISPLAY TCPIP operator command to display the TCP/IP stack name as shown in the following example:
![]() |
agent.yml
Parameter | Description | Required | Default |
---|---|---|---|
port | The agent’s listening port for server HTTPS communication | No | 9999 |
keystore | An agent keystore in JKS format which contains only one key entry to be used by the agent for incoming HTTPS requests. See Optional: Using the organization's certification authority. | No | agent.jks |
keystore_password | The provided keystore password | No | agent |
truststore | An agent truststore in JKS format which the agent will use to trust incoming HTTPS requests. See Optional: Using the organization's certification authority. | No | keystore value |
truststore_password | The provided truststore password | No | keystore_password value |
Configure the object storage settings by providing the following parameters:
Parameter | Description | Required | Value |
---|---|---|---|
objstore.endpoint.url | The URL of the object storage (Should start with “http://” or “https://”) | yes | default: none |
objstore.endpoint.userid | Access key to object storage | yes | default: none |
objstore.endpoint.password | Secret key to object storage | yes | default: none |
objstore.resources.container.name | Container/bucket name | yes | default: model9-data |
objstore.endpoint.api.id | The object storage API name | no | default: s3 Amazon AWS: aws-s3 Microsoft Azure: azureblob |
objstore.endpoint.api.s3.v4signatures | When using object storage that uses V4 signatures, set this parameter to ‘true’, in addition to api.id: s3 | no | default: false Cohesity: true HCP-CS: true |
objstore.endpoint.no.verify.ssl | When using the HTTPS protocol, set to ‘true’ to avoid SSL certificate verifications* | no | default: true |
* To enable trust validation on the object storage certificate, use the objstore.endpoint.no.verify.ssl parameter and set it to ‘false’.
The agent.yml file may contain additional parameters, setting them will override the defaults:
Parameter Name | Description | Default value |
---|---|---|
objstore.endpoint.shared.archive.volume | Archived volume serial name | M9ARCH |
objstore.functionality.backup. threadpool.size | Number of parallel threads for backup and archive runs. Increasing this parameter requires verifying that sufficient java heap space is defined for the agent . | 20 |
load.balancing.group.name | A group of agents that can share the work of a single policy. | &SYSPLEX |
restore.progress.interval. percentage | Progress interval by percentage | 10 |
restore.progress.minimum.size.mb | The minimum size in MB of a backup/archive action at which point the Agent starts reporting recall/restore progress | 150 |
objstore.resources.complex.id | Defines the resources the agent can process. | “group- &SYSPLEX” |
lifecycle.deleteExpired.gdg.minimumDays | The minimum number of days that a data set will be kept after being archived. After the number of minimum days has passed, the data set will be checked whether it is eligible for deletion. | 7 |
lifecycle.deleteExpired.gdg.intervalDays | After the minimum number of days has passed, the data set will be checked whether it is eligible for deletion. The data set will be checked every X days. | 4 |
cli.functionality.backup.compression | The compression type used for CLI backup execution. | gzip |
cli.functionality.archive.compression | The compression type used for CLI archive execution. | gzip |
import.allocate_use_expdt_98000 | Whether or not to add the EXPDT(98000) to the tape allocations. For CA-1 import support change to: true | false |
import.allocate_label_type | Which label type should be used by the tape allocation. For CA-1 import support change to: blp | sl |
Resource complex and load balancing parameters
Agents sharing the same resource complex can all interact with the same resources. Actions such as backup/archive/recall/restore/list are all dependent on the resource complex parameter. Agents not sharing the same resource complex will not be able to list/restore/recall backups/archives created on a different resource complex. By default, all agents in the same Sysplex share the same resource complex. To update the default value of the resource complex, add the parameter to your agent.yml configuration file with a new value. Within a resource complex, you can define one or more load balancing groups. See the Administrator and User Guide for more details.
Note
An agent can only access resources backed up or archived to the complex ID defined in its configuration.
Step 11: Update the Model9 agent license
Update the license file at /usr/lpp/model9/conf/model9-licenses.txt. The file can accommodate several licenses, for example, the license of all the LPARS in the Sysplex and the Disaster Recovery site’s license.
Each license should be written on a separate line.
Comments should be preceded with a “#” sign.
When multiple licenses are specified in the file, the most fitting license will be used, according to the following rules:
The CPUID correlates with the CPUID of the executing LPAR. This is mandatory.
From the licenses that correlate with the CPUID of the executing LPAR, the one with the latest expiration date will be used.
The license details to be used will be displayed in the agent’s log and in the UI. In case the agent could not find a valid license, an error message will be displayed as a WTO. The full error message will be available in STDOUT.
Optional: Secure the agent - server communication
By default, the agent accepts requests from trusted clients using a self-signed certificate generated by Model9. In a production environment, it is recommended to replace the certificate with a signed organizational CA certificate.
Before you start, verify that the server has:
A JKS type truststore containing the CA chain certificates.
A PKCS12 type keystore containing the server’s private key and its certificate, signed by the root CA.
Verify that the agent has:
A JKS type truststore containing the CA chain certificates (same as the server’s).
A JKS type keystore containing the agent’s private key and its certificate, signed by the root CA.
To replace the default certificates, run the following commands from $MODEL9_HOME/keys path:
Create the certificate requests: - for the server’s certificate:
openssl genrsa -out server.key 2048 openssl req -new -key server.key -out server.csr
- for the agent’s certificate:
openssl genrsa -out agent.key 2048 openssl req -new -key agent.key -out agent.csr
Send the two certificate request files to your local CA for signing.
Request the CA chain certificate as well.
Import the certificates on the server / any Linux:
For both the server and the agent: import the CA key into the JKS - Agent/Server truststore:
keytool -keystore rootCA.jks -import -v -alias root -file rootCA.cer -keypass model9
For the server: import the certificate and private key into the PKCS12 store - Server keystore:
#OpenSSL to create p12 store openssl pkcs12 -export -in server.cer -inkey server.key -name agent -out server.p12
For the agent: import the certificate and private key into the Agent keystore:
#OpenSSL to create p12 store openssl pkcs12 -export -in agent.cer -inkey agent.key -name agent -out agent.p12 #Import the p12 store into JKS keytool -importkeystore -deststorepass model9 -destkeystore agent.jks -srckeystore agent.p12 -srcstoretype PKCS12
The server:
Upload the server.p12 file to the server’s $MODEL9_HOME/keys path.
Update the server’s configuration file on the Linux system (model9-local.yml file):
keystore -> filename – should point to the server.p12 (for example: MODEL9_HOME/keys/server.p12)
truststore -> filename – should point to the rootCA.jks (for example: MODEL9_HOME/keys/rootCA.jks)
The agent:
upload the agent.jks and rootCA.jks files to the Agent’s conf path on the z/OS (binary mode).
Update the agent’s configuration file on the z/OS (agent.yml file):
keystore – should point to the agent.jks file (for example: ../conf/agent.jks)
truststore – should point to the rootCA.jks file (for example: ../conf/rootCA.jks)
Step 12: Install automatic recall
The Model9 Automatic recall Hook allows recall of archived files from within z/OS in a transparent manner. To install the automatic recall hook:
Update the PROGxx configuration by adding the following statements:
APF ADD DSNAME(SYS2.MODEL9.V181.LOADLIB) SMS LPA ADD DSNAME(SYS2.MODEL9.V181.LOADLIB) MOD(ZM9CPTN) LPA ADD DSNAME(SYS2.MODEL9.V181.LOADLIB) MOD(ZM9S26X) EXIT ADD EXITNAME(ZM9P_S026) MOD(ZM9S26X) PARM('M9ARCH')
Apply the changes using the SET PROG=xx operator command from any console:
SET PROG=xx
Note
The hook and exit must be loaded to the Dynamic LPA in order for them to function correctly, do not use MLPA or PLPA to load the modules.
Copy the M9HOOK and M9UNHOOK from the SAMPLIB PDS to a local PROCLIB member. Customize M9HOOK accordingly and activate it using the following command:
S M9HOOK
Note
Add the M9HOOK JCL to your standard IPL process.
If an existing Model9 Hook has been uninstalled, then a new copy of the ZM9CPTN module must be loaded to the Dynamic LPA using the following command:
SETPROG LPA ADD DSNAME=SYS2.MODEL9.V181.LOADLIB MOD=ZM9CPTN
When using another data management product together with Model9 Cloud Data Manager, add the following DD statement to the other product’s procedure to avoid collisions:
//ZM9$NORC DD DUMMY
Restart the address space after applying the DD.
Step 13: Life cycle management
The Model9 life cycle management JCL is responsible for automatically deleting archived data sets that have expired. A sample JCL, M9LIFECY, can be found in the Model9 SAMPLIB PDS.
Grant your site scheduler permission for surrogate access to the M9USER:
RDEFINE SURROGAT M9USER.SUBMIT UACC(NONE) PERMIT M9USER.SUBMIT CLASS(SURROGAT) ID(<scheduler>) ACC(READ) SETR REFRESH RACLIST(SURROGAT)
Update the life cycle management JCL:
Update
Value description
DD STEPLIB
Model9 installation LOADLIB
PWD environment variable
Model9 agent’s installation path
Submit the life cycle management JCL in the simulate mode: Set --simulate to ‘yes’. The life cycle management JCL will list all the data sets that would have been deleted without actually deleting them.
Submit the life cycle management JCL in the initial mode: Set --simulate to ‘no’ or remove. The life cycle management JCL will scan all archived data sets with expiration dates in the past - up until today’s date. The next time the life cycle will run, it will resume where the previous process ended.
Note
Life cycle management JCL should be scheduled daily via your site scheduler.
Step 14: Start the Model9 agent
Start the agent from any console by issuing the following command:
S M9AGENT
Verify that the agent was started successfully. The following messages should appear:
ZM91002I MODEL9 BACKUP AGENT VERSION 1.8.1 INITIALIZING
ZM91000I MODEL9 BACKUP AGENT INITIALIZED
Optional: Configure the Remote Server API M9SAPI
The optional Remote Server API feature allows running of Model9 policies from within z/OS using standard JCL. To enable the Remote Server API feature:
Copy the sample JCL PROC M9SAPI from the SAMPLIB PDS to a local PROCLIB file and edit the procedure according to the instructions in the file. The JCL M9SAPIJ sample can be used to run a policy.
The Remote Server API uses HTTPS to communicate with the server. If a trusted certificate is not defined for the server, set VRFYCERT=NO in the M9SAPI procedure to skip certificate validations.
If a valid certificate is defined, make sure the user running the M9APIJ has the correct SAF keyring defined to enable validation of the server certificate. The SAPI_KEYRING_NAME setting defined in the sample model9-stdenv.sh configuratiuon can be used to specify the keyring name.
Note
Communications between the Remote Server API and the Model9 Server are encrypted regardless of the VRFYCERT value.
When verifying the certificate used by the Remote Server API, it must be defined for a Resource Access Control Facility (RACF) using a keyring.
Add a digital certificate to RACF, and create a keyring using the following commands:
RACDCERT ADD('p12-cert-dataset-name') CERTAUTH TRUST WITHLABEL('label') PASSWORD('pkcs12-password') RACDCERT ADDRING(ring-name) ID(username) SETROPTS RACLIST(DIGTRING) REFRESH //if DIGTRING class is RACFLISTed RACDCERT CONNECT(CERTAUTH LABEL('label') RING(ring-name) SETROPTS RACLIST(DIGTCERT, DIGTRING) REFRESH //if DIGTRING & DIGTRING class is RACFLISTed
If a keyring is used, verify that the user running the Remote Server API JCL has the following permissions:
IRR.DIGTCERT.LISTRING CLASS(FACILITY) ACCESS(READ)
Optional: Configure ISPF to use M9ARCH as ML2 default volume
ISPF allows the definition of only one Level 2 migration volume. The default migration volume is MIGRAT. In order to use Model9 automatic recall function, the default volume must be set to M9ARCH, using the ISPF Configuration utility. Failing to change the migration volume will result in the following errors in ISPF screen 3.4:
Command | Situation | Resolution |
---|---|---|
Browse / Edit / View | Automatic recall will be rightfully triggered but the screen will continue to display M9ARCH as the VOLSER, instead of the DASD. The message "Tape not supported" will appear on the top right of the ISPF screen. | Enter REFRESH on the command line. The screen will be refreshed, showing the correct VOLSER, and the file will be available. |
LISTC or performing a REXX | Automatic recall will be unnecessarily triggered and the data set will be recalled | Use “TSO LISTC” instead of typing LISTC next to the data set name |
Note
Performing this step will cause any other auto-recall software to suffer from these errors
To change the default ISPF settings:
Start the ISPF Configuration utility (TSO ISPCCONF) and select the site defined in the configuration table.
Select Option 1 to modify the existing table.
Change the “Volume for Migrated Data Sets name” to M9ARCH - the default Model9 archive volume name.
Save the settings.
Review the newly created configuration table file.
Either create and install the SMP/E USERMOD containing the new configuration table or create a load module that resides on a shared library containing the ISPF settings.