Cloud Data Manager

Installing the Model9 agent

Prerequisites

License key

Obtain your license key from Model9 by opening a “new license request” in the Model9 service portal: https://model9.atlassian.net/servicedesk/customer/portals

The output of the z/OS command “D M=CPU” is required for obtaining a new license.

Java version

Verify that the z/OS Java is version 8 64-bit SR5 FP16 or above. Verify the Java version and change the Java installation directory using the following commands:

cd /usr/lpp/java/J8.0_64/bin
./java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.27 - pmz6480sr5fp27-20190104_01(SR5 FP16))
IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20181219_405297 (JIT enabled, AOT enabled)
OpenJ9 - 3f2d574
OMR - 109ba5b
IBM - e2996d1)
JCL - 20190104_01 based on Oracle jdk8u191-b26

UTC date and time

The object storage protocol requires the z/OS USS and object storage UTC date and time to match. Use the following USS command to display and verify the UTC date and time:

date -u

Note

If the UTC date and time do not match, the agent will fail with an HTTP error code 403 while trying to connect to the object storage.

JZOS setup

The Agent is invoked using the JZOS Batch Java Launcher which is a component of the IBM JVM for z/OS. To verify that the JZOS is correctly configured to run the Agent, check the JZOS JCL procedures and make sure member JVMPRC86 is located in a site standard JES PROCLIB. If any of the JZOS resources are missing, copy them from the Java installation to your installation:

//M9JAVAP JOB ACCT#,SYSPROG,TIME=NOLIMIT,REGION=0M,
// NOTIFY=&SYSUID,MSGLEVEL=(1,1),MSGCLASS=X
//COPYFILE EXEC PGM=IKJEFT01
//IN DD PATH='/usr/lpp/java/J8.0_64/mvstools/samples/jcl/JVMPRC86'
//OUT DD DISP=SHR,DSN=SYS1.PROCLIB(JVMPRC86)
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
OCOPY INDD(IN) OUTDD(OUT) TEXT
/*
//

Note

For more details about JZOS setup, refer to the JZOS Installation and User Guide: http://publibfp.dhe.ibm.com/epubs/pdf/ajvc0120.pdf

Step 1: Allocate and mount the ZFS filesystem

Create a dedicated ZFS filesystem for the Model9 Agent installation and mount point for the new ZFS, as demonstrated in the following JCL. Edit the JCL using the site’s standard naming convention for the mount point, ZFS and volume serial number if it is not managed by the SMS.

//M9AGTZFS JOB ACCT#,SYSPROG,TIME=NOLIMIT,REGION=0M,
// NOTIFY=&SYSUID,MSGLEVEL=(1,1),MSGCLASS=X
//* Create a Model9 installation directory 
//CREATE   EXEC   PGM=BPXBATCH,      
// PARM='SH mkdir -p /usr/lpp/model9'
//* Define and format a new zFS for Model9 
//DEFINE   EXEC   PGM=IDCAMS
//SYSPRINT DD     SYSOUT=*
//SYSIN    DD     *
 DEFINE CLUSTER (NAME(SYS2.MODEL9.ZFS) -
 VOLUMES(xxxxxx) -
 LINEAR CYL(200 50))
/*
//FORMAT   EXEC   PGM=IOEAGFMT,REGION=0M,
// PARM=('-aggregate SYS2.MODEL9.ZFS -compat')
//SYSPRINT DD     SYSOUT=*
//STDOUT   DD     SYSOUT=*
//STDERR   DD     SYSOUT=*
//* Mount the newly defined zFS to the Model9 installation directory
//MOUNT    EXEC   PGM=IKJEFT01,DYNAMNBR=10
//SYSTSPRT DD     SYSOUT=*
//SYSTSIN  DD     *
 MOUNT FILESYSTEM('SYS2.MODEL9.ZFS') +
 MOUNTPOINT('/usr/lpp/model9') +
TYPE(ZFS) MODE(RDWR) /* AUTOMOVE */
//

Note

The parameters for the IOEAGFMT utility and the mount commands are case sensitive.

To automatically and permanently mount the ZFS after an IPL, add the mount command to the BPXPRMxx as shown in the following example:

MOUNT FILESYSTEM('SYS2.MODEL9.ZFS')
TYPE(ZFS)
MODE(RDWR)
MOUNTPOINT('/usr/lpp/model9') /* AUTOMOVE */

Note

For a Sysplex environment, remove the comment marks from the term AUTOMOVE in the JCL above to enable file system migration between Sysplex members.

Step 2: Upload the Model9 agent TAR file to the mainframe

Use an FTP utility to upload the Model9 agent’s installation tar file to the model9 directory created in the previous step. Use Passive Mode if supported by the FTP client. The tar file must be uploaded in binary mode as shown in the following example:

$ ftp mf-lp1
Connected to mf-lp1.
220-FTPD1 IBM FTP CS V2R2 at mf-lp1, 06:20:40 on 2017-02-23.
220 Connection will not timeout.
Name (mf-lp1:m9user): m9user
331 Send password please.
Password:
230 M9U is logged on. Working directory is "M9U.".
Remote system type is MVS.
ftp> cd /usr/lpp/model9/
250 HFS directory /usr/lpp/model9/ is the current working directory
ftp> bin
200 Representation type is Image
ftp> put model9-v1.8.1_build_48c2c57f-agent.tar
local: model9-v1.8.1_build_48c2c57f-agent.tar remote: model9-v1.8.1_build_48c2c57f-agent.tar
229 Entering Extended Passive Mode (|||1026|)
125 Storing data set /usr/lpp/model9/model9-v1.8.1_build_48c2c57f-agent.tar
250 Transfer completed successfully.
4483584 bytes sent in 00:02 (1.95 MiB/s)
ftp> quit
221 Quit command received. Goodbye.

Step 3: Extract the agent files from the TAR file

Use the tar command in the z/OS UNIX shell to extract the agent files. During extraction, all agent files are saved in the same directory and can only be run by users who have the following permissions in the FACILITY class:

  • BPX.FILEATTR.APF ACC(READ)

  • BPX.FILEATTR.PROGCTL ACC(READ)

  • BPX.FILEATTR.SHARELIB ACC(READ)

The directory name usually includes the agent’s release number. It is recommended to create an alias that does not include the release number for the directory. This makes future upgrades more transparent, as shown in the following example:

TSO OMVS
su
cd /usr/lpp/model9/
tar -xpf model9-v1.8.1_build_48c2c57f-agent.tar
# define the alias name “agent”
ln -s model9-v1.8.1_build_48c2c57f-agent agent

Step 4: Create and populate the configuration directory

Create and copy the Model9 sample configuration directory using the following commands:

cd /usr/lpp/model9
mkdir conf
cp agent/sampleConf/* conf/

Step 5: Create a program-controlled version of PAX

The Model9 z/OS UNIX files backup uses PAX. This functionality requires creation of a program-controlled version of PAX. Use the following commands:

cd /usr/lpp/model9
mkdir bin
cp -p /bin/pax bin
# Set program-control flag
extattr +p bin/pax

Step 6: Copy the Model9 libraries from USS to PDS

Edit the JCL CPY#PDS located in /usr/lpp/model9/agent/installation/ to create the Model9 LOADLIB, SAMPLIB and EXEC PDS files. Submit the JCL.

Step 7: Install the Model9 Command Line Interface

The Model9 CLI Command Line Interface provides an interface for issuing Model9 commands from TSO / JCL. Install the feature by creating a listener directory under the agent’s main path:

cd /usr/lpp/model9/
mkdir listener
chmod 777 listener
chmod +t listener

Customize the M9CLI rexx in the EXEC PDS to match installation standards:

fifodir = "/usr/lpp/model9/listener"
loaddir = "SYS2.MODEL9.V181.LOADLIB"

Note

Copy the M9CLI EXEC to your site’s local EXEC library.

Step 8: Define the agent to RACF

Use the sample JCL M9USERST in the SAMPLIB PDS to define the security settings required by the agent. M9USERST permits discrete profiles. If your site uses generic profiles, update the JCL accordingly. If your site uses unprotected resources, no further action is needed.

Note

Review and update the JCL to match local site standards. The SHARED parameter is specified by default. The SHARED.IDS permission is required to run the JCL.

Submit the JCL.

Step 9: Customize the Model9 agent start procedure

Copy the sample JCL M9AGENT from the SAMPLIB PDS to a local PROCLIB member. Update the JCL:

Update

Description

DD STEPLIB

Model9 installation LOADLIB

PWD environment variable

Model9 agent’s installation path

CONF_HOME environment variable*

Model9 agent’s configuration directory path

*CONF_HOME can be used for activating more than one agent in the same LPAR. The parameter will allow the agents to use the same Model9 installation files and libraries, but have a different configuration directory. The recommendation is to have one agent per LPAR, and to have all agents in the same GRS-complex point to the same Model9 complex. However, additional agents in the same LPAR may be required if using a sub-plex, having both development and production environments or pointing the agents to different cloud storage. CONF_HOME must precede the stdenv-main.sh statement. The following is a sample use of the parameter:

//STDENV DD *
export PWD=/usr/lpp/model9/agent
export CONF_HOME=$PWD/../conf
export ENV=agent
. $PWD/scripts/stdenv-main.sh
//

Note

Add the agent procedure to the system’s startup process.

Step 10: Update the Model9 agent configuration

Before starting the agent, update the following configuration files located in /usr/lpp/model9/conf:

model9-stdenv.sh

Parameter

Description

Required

Default

JAVA_HOME

64-bit java home path

Yes

None

TIME_ZONE

Java time zone, e.g. America/New_York

Yes

None

TCPIP_NAME

TCP/IP stack used by the agent *

No

TCPIP

AGENT_MEMORY

Allocated Java heap space for the agent. 1 GB is the minimum value.

No

1g

LIFE_CYCLE_MEMORY

Allocated Java heap space for life cycle management. 256 MB is the minimum value.

No

256m

SAPI_MEMORY

Allocated Java heap space for the server API (M9SAPI). 56 MB is the minimum value.

No

56m

SAPI_NETWORK_DEBUG

Enable network level debug for server API (M9SAPI). Expected values: “true” | “false”

No

false

SAPI_KEYRING_NAME

The SAF keyring name to be used as a truststore during server API certificate validation. if specified, the keyring will be searched at: &SYSUID/<KeyRingName>

No

None

* If running in an environment with multiple TCP/IP stacks, use the DISPLAY TCPIP operator command to display the TCP/IP stack name as shown in the following example:

image9.png

agent.yml

Parameter

Description

Required

Default

port

The agent’s listening port for server HTTPS communication

No

9999

keystore

An agent keystore in JKS format which contains only one key entry to be used by the agent for incoming HTTPS requests. See Optional: Using the organization's certification authority.

No

agent.jks

keystore_password

The provided keystore password

No

agent

truststore

An agent truststore in JKS format which the agent will use to trust incoming HTTPS requests. See Optional: Using the organization's certification authority.

No

keystore value

truststore_password

The provided truststore password

No

keystore_password value

Configure the object storage settings by providing the following parameters:

Parameter

Description

Required

Value

objstore.endpoint.url

The URL of the object storage (Should start with “http://” or “https://”)

yes

default: none

objstore.endpoint.userid

Access key to object storage

yes

default: none

objstore.endpoint.password

Secret key to object storage

yes

default: none

objstore.resources.container.name

Container/bucket name

yes

default: model9-data

objstore.endpoint.api.id

The object storage API name

no

default: s3

Amazon AWS: aws-s3

Microsoft Azure: azureblob

objstore.endpoint.api.s3.v4signatures

When using object storage that uses V4 signatures, set this parameter to ‘true’, in addition to api.id: s3

no

default: false Cohesity: true HCP-CS: true

objstore.endpoint.no.verify.ssl

When using the HTTPS protocol, set to ‘true’ to avoid SSL certificate verifications*

no

default: true

* To enable trust validation on the object storage certificate, use the objstore.endpoint.no.verify.ssl parameter and set it to ‘false’.

The agent.yml file may contain additional parameters, setting them will override the defaults:

 

Parameter Name

Description

Default value

objstore.endpoint.shared.archive.volume

Archived volume serial name

M9ARCH

objstore.functionality.backup. threadpool.size

Number of parallel threads for backup and archive runs. Increasing this parameter requires verifying that sufficient java heap space is defined for the agent .

20

load.balancing.group.name

A group of agents that can share the work of a single policy.

&SYSPLEX

restore.progress.interval.

percentage

Progress interval by percentage

10

restore.progress.minimum.size.mb

The minimum size in MB of a backup/archive action at which point the Agent starts reporting recall/restore progress

150

objstore.resources.complex.id

Defines the resources the agent can process.

“group- &SYSPLEX”

lifecycle.deleteExpired.gdg.minimumDays

The minimum number of days that a data set will be kept after being archived. After the number of minimum days has passed, the data set will be checked whether it is eligible for deletion.

7

lifecycle.deleteExpired.gdg.intervalDays

After the minimum number of days has passed, the data set will be checked whether it is eligible for deletion. The data set will be checked every X days.

4

cli.functionality.backup.compression

The compression type used for CLI backup execution.

gzip

cli.functionality.archive.compression

The compression type used for CLI archive execution.

gzip

import.allocate_use_expdt_98000

Whether or not to add the EXPDT(98000) to the tape allocations.

For CA-1 import support change to: true

false

import.allocate_label_type

Which label type should be used by the tape allocation.

For CA-1 import support change to: blp

sl

Resource complex and load balancing parameters

Agents sharing the same resource complex can all interact with the same resources. Actions such as backup/archive/recall/restore/list are all dependent on the resource complex parameter. Agents not sharing the same resource complex will not be able to list/restore/recall backups/archives created on a different resource complex. By default, all agents in the same Sysplex share the same resource complex. To update the default value of the resource complex, add the parameter to your agent.yml configuration file with a new value. Within a resource complex, you can define one or more load balancing groups. See the Administrator and User Guide for more details.

Note

An agent can only access resources backed up or archived to the complex ID defined in its configuration.

Step 11: Update the Model9 agent license

Update the license file at /usr/lpp/model9/conf/model9-licenses.txt. The file can accommodate several licenses, for example, the license of all the LPARS in the Sysplex and the Disaster Recovery site’s license.

  • Each license should be written on a separate line.

  • Comments should be preceded with a “#” sign.

When multiple licenses are specified in the file, the most fitting license will be used, according to the following rules:

  • The CPUID correlates with the CPUID of the executing LPAR. This is mandatory.

  • From the licenses that correlate with the CPUID of the executing LPAR, the one with the latest expiration date will be used.

The license details to be used will be displayed in the agent’s log and in the UI. In case the agent could not find a valid license, an error message will be displayed as a WTO. The full error message will be available in STDOUT.

Optional: Secure the agent - server communication

By default, the agent accepts requests from trusted clients using a self-signed certificate generated by Model9. In a production environment, it is recommended to replace the certificate with a signed organizational CA certificate.

Before you start, verify that the server has:

  1. A JKS type truststore containing the CA chain certificates.

  2. A PKCS12 type keystore containing the server’s private key and its certificate, signed by the root CA.

Verify that the agent has:

  1. A JKS type truststore containing the CA chain certificates (same as the server’s).

  2. A JKS type keystore containing the agent’s private key and its certificate, signed by the root CA.

To replace the default certificates, run the following commands from $MODEL9_HOME/keys path:

  1. Create the certificate requests: - for the server’s certificate:

    openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr

    - for the agent’s certificate:

    openssl genrsa -out agent.key 2048
openssl req -new -key agent.key -out agent.csr

  2. Send the two certificate request files to your local CA for signing.

  3. Request the CA chain certificate as well.

  4. Import the certificates on the server / any Linux:

    For both the server and the agent: import the CA key into the JKS - Agent/Server truststore:

    keytool -keystore rootCA.jks -import -v -alias root -file rootCA.cer -keypass model9

    For the server: import the certificate and private key into the PKCS12 store - Server keystore:

    #OpenSSL to create p12 store
openssl pkcs12 -export -in server.cer -inkey server.key -name agent -out server.p12

    For the agent: import the certificate and private key into the Agent keystore:

    #OpenSSL to create p12 store
openssl pkcs12 -export -in agent.cer -inkey agent.key -name agent -out agent.p12
#Import the p12 store into JKS
keytool -importkeystore -deststorepass model9 -destkeystore agent.jks -srckeystore agent.p12 -srcstoretype PKCS12

  5. The server:

    1. Upload the server.p12 file to the server’s $MODEL9_HOME/keys path.

    2. Update the server’s configuration file on the Linux system (model9-local.yml file):

      • keystore -> filename – should point to the server.p12 (for example: MODEL9_HOME/keys/server.p12)

      • truststore -> filename – should point to the rootCA.jks (for example: MODEL9_HOME/keys/rootCA.jks)

  6. The agent:

    1. upload the agent.jks and rootCA.jks files to the Agent’s conf path on the z/OS (binary mode).

    2. Update the agent’s configuration file on the z/OS (agent.yml file):

      • keystore – should point to the agent.jks file (for example: ../conf/agent.jks)

      • truststore – should point to the rootCA.jks file (for example: ../conf/rootCA.jks)

Step 12: Install automatic recall

The Model9 Automatic recall Hook allows recall of archived files from within z/OS in a transparent manner. To install the automatic recall hook:

Update the PROGxx configuration by adding the following statements:

APF ADD DSNAME(SYS2.MODEL9.V181.LOADLIB) SMS
LPA ADD DSNAME(SYS2.MODEL9.V181.LOADLIB) MOD(ZM9CPTN)
LPA ADD DSNAME(SYS2.MODEL9.V181.LOADLIB) MOD(ZM9S26X)
EXIT ADD EXITNAME(ZM9P_S026) MOD(ZM9S26X) PARM('M9ARCH')

Apply the changes using the SET PROG=xx operator command from any console:

SET PROG=xx

Note

The hook and exit must be loaded to the Dynamic LPA in order for them to function correctly, do not use MLPA or PLPA to load the modules.

Copy the M9HOOK and M9UNHOOK from the SAMPLIB PDS to a local PROCLIB member. Customize M9HOOK accordingly and activate it using the following command:

S M9HOOK

Note

Add the M9HOOK JCL to your standard IPL process.

If an existing Model9 Hook has been uninstalled, then a new copy of the ZM9CPTN module must be loaded to the Dynamic LPA using the following command:

SETPROG LPA ADD DSNAME=SYS2.MODEL9.V181.LOADLIB MOD=ZM9CPTN

When using another data management product together with Model9 Cloud Data Manager, add the following DD statement to the other product’s procedure to avoid collisions:

//ZM9$NORC DD DUMMY

Restart the address space after applying the DD.

Step 13: Life cycle management

The Model9 life cycle management JCL is responsible for automatically deleting archived data sets that have expired. A sample JCL, M9LIFECY, can be found in the Model9 SAMPLIB PDS.

  1. Grant your site scheduler permission for surrogate access to the M9USER:

    RDEFINE SURROGAT M9USER.SUBMIT UACC(NONE)
PERMIT M9USER.SUBMIT CLASS(SURROGAT) ID(<scheduler>) ACC(READ)
SETR REFRESH RACLIST(SURROGAT)

  2. Update the life cycle management JCL:

    Update

    Value description

    DD STEPLIB

    Model9 installation LOADLIB

    PWD environment variable

    Model9 agent’s installation path

  3. Submit the life cycle management JCL in the simulate mode: Set --simulate to ‘yes’. The life cycle management JCL will list all the data sets that would have been deleted without actually deleting them.

  4. Submit the life cycle management JCL in the initial mode: Set --simulate to ‘no’ or remove. The life cycle management JCL will scan all archived data sets with expiration dates in the past - up until today’s date. The next time the life cycle will run, it will resume where the previous process ended.

Note

Life cycle management JCL should be scheduled daily via your site scheduler.

Step 14: Start the Model9 agent

Start the agent from any console by issuing the following command:

S M9AGENT

Verify that the agent was started successfully. The following messages should appear:

ZM91002I MODEL9 BACKUP AGENT VERSION 1.8.1 INITIALIZING
ZM91000I MODEL9 BACKUP AGENT INITIALIZED

Optional: Configure the Remote Server API M9SAPI

The optional Remote Server API feature allows running of Model9 policies from within z/OS using standard JCL. To enable the Remote Server API feature:

  1. Copy the sample JCL PROC M9SAPI from the SAMPLIB PDS to a local PROCLIB file and edit the procedure according to the instructions in the file. The JCL M9SAPIJ sample can be used to run a policy.

  2. The Remote Server API uses HTTPS to communicate with the server. If a trusted certificate is not defined for the server, set VRFYCERT=NO in the M9SAPI procedure to skip certificate validations.

  3. If a valid certificate is defined, make sure the user running the M9APIJ has the correct SAF keyring defined to enable validation of the server certificate. The SAPI_KEYRING_NAME setting defined in the sample model9-stdenv.sh configuratiuon can be used to specify the keyring name.

    Note

    Communications between the Remote Server API and the Model9 Server are encrypted regardless of the VRFYCERT value.

  4. When verifying the certificate used by the Remote Server API, it must be defined for a Resource Access Control Facility (RACF) using a keyring.

    1. Add a digital certificate to RACF, and create a keyring using the following commands:

      RACDCERT ADD('p12-cert-dataset-name') CERTAUTH TRUST WITHLABEL('label') PASSWORD('pkcs12-password')
RACDCERT ADDRING(ring-name) ID(username)
SETROPTS RACLIST(DIGTRING) REFRESH //if DIGTRING class is RACFLISTed
RACDCERT CONNECT(CERTAUTH LABEL('label') RING(ring-name)
SETROPTS RACLIST(DIGTCERT, DIGTRING) REFRESH //if DIGTRING & DIGTRING class is RACFLISTed

    2. If a keyring is used, verify that the user running the Remote Server API JCL has the following permissions:

      IRR.DIGTCERT.LISTRING CLASS(FACILITY) ACCESS(READ)

Optional: Configure ISPF to use M9ARCH as ML2 default volume

ISPF allows the definition of only one Level 2 migration volume. The default migration volume is MIGRAT. In order to use Model9 automatic recall function, the default volume must be set to M9ARCH, using the ISPF Configuration utility. Failing to change the migration volume will result in the following errors in ISPF screen 3.4:

Command

Situation

Resolution

Browse / Edit / View

Automatic recall will be rightfully triggered but the screen will continue to display M9ARCH as the VOLSER, instead of the DASD. The message "Tape not supported" will appear on the top right of the ISPF screen.

Enter REFRESH on the command line. The screen will be refreshed, showing the correct VOLSER, and the file will be available.

LISTC or performing a REXX

Automatic recall will be unnecessarily triggered and the data set will be recalled

Use “TSO LISTC” instead of typing LISTC next to the data set name

Note

Performing this step will cause any other auto-recall software to suffer from these errors

To change the default ISPF settings:

  1. Start the ISPF Configuration utility (TSO ISPCCONF) and select the site defined in the configuration table.

  2. Select Option 1 to modify the existing table.

  3. Change the “Volume for Migrated Data Sets name” to M9ARCH - the default Model9 archive volume name.

  4. Save the settings.

  5. Review the newly created configuration table file.

Either create and install the SMP/E USERMOD containing the new configuration table or create a load module that resides on a shared library containing the ISPF settings.

See also: