Cloud Data Manager

Step-by-step Installation

This guide will instruct you how to set up and invoke the transform service of the Model 9 Cloud Data Gateway from the mainframe, using JCL. The service will transform a Model9 data set backup copy / archive into a readable file in the cloud. Once transformed, the readable file can be accessed directly or via data analytics tools.

If using the Model 9 Cloud Data Gateway as an on-premises service, refer to Step-by-step deploymentguide before continuing to the steps on this guide.Step-by-step deployment

Note

The transform service is invoked as SaaS by using the url:

https://us-east-1.model9api.io/

Step 1: Set up

Verify Model 9 Cloud Backup and Recovery for z/OS is installed

Model9 is responsible to deliver the data set from the mainframe to the cloud / on-premises storage. The data set is delivered as a backup copy or an archive, and provides the input to the transform service.

Download z/OS cURL

This free tool will allow you to invoke the transform service from z/OS. If cURL is not installed under /usr/bin, edit line 4 and add the path where the cURL module resides.

Step 2: Copy the script

Copy the following script to /usr/lpp/model9/transformService.sh:

#!/bin/sh
json=$1
url='https://us-east-1.model9api.io/transform'
export PATH=/usr/bin:/bin
export _EDC_ADD_ERRNO2=1
cnvto="$(locale codeset)"
headers="Content-Type:application/json"
echo "Running Model9 transform service"
output=$(curl -H "$headers" -s -k -X POST --data "$json" $url)
if ! [ -z "$output" ]; then echo "Transform ended with the following output:" # If the answer is in ASCII then convert to EBCDIC firstChar=$(echo $output | cut -c1) if [ "$firstChar" = "#" ]; then convOutput="$(echo $output | iconv -f ISO8859-1 -t $cnvto)" else convOutput=$output fi echo "$convOutput" fi
status=$(echo $convOutput | tr -s " " | cut -d, -f1 | cut -d" " -f3) echo "Transform ended with status: $status"
if [ "$status" = '"OK"' ];then exit 0
else if [ "$status" = '"WARNING"' ]; then exit 4
else exit 8
fi
fi
Step 3: Copy the JCL

Copy the following JCL to a local library, update the JOBCARD according to your site standards:

//M9TRNSFM JOB 'ACCT#',REGION=0M,CLASS=A,NOTIFY=&SYSUID
//EXTRACT EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDPARM DD *
SH /usr/lpp/model9/transformService.sh
// DD *,SYMBOLS=EXECSYS
'{ "input": { "name" : "<DATA-SET>", "complex": "group-&SYSPLEX", "archive": "<false|true>" }, "output": { "prefix" : "/transform/&LYR4/&LMON/&LDAY", "compression" : "none", "format" : "text" }, "source": { "url" : "<URL>", "api" : "<API>", "bucket" : "<BUCKET>", "user" : "<USER>", "password": "<PASSWORD>" }
}'
/*
//
Step 4: Customize the JCL

Update the object storage details

Copy the following object storage variables from the Model9 agent configuration file:

  • <URL>

  • <API>

  • <BUCKET>

  • <USER>

  • <PASSWORD>

Update the complex name

The “complex” name represents the group of resources that the Model9 agent can access. By default, this group is named group-<SYSPLEX> and it is shared by all the agents in the same sysplex. The transform JCL specifies the default, using the z/OS system symbol “&SYSPLEX”.

Note

If the default was kept for “complex” in the Model9 agent configuration file, no change is needed

If the “complex” name was changed in the Model9 agent configuration file, change the “complex” in the JCL accordingly.

Update the transform prefix

By default, the JCL will create a transformed copy of your input data set, in the same bucket, with the prefix: /transform/&LYR4/&LMON/&LDAY. The prefix is using the following z/OS system symbols:

  • &LYR4 - The year in 4 digits, e.g. 2019

  • &LMON - The month in 2 digits, e.g. 08

  • &LDAY - The day in the month in 2 digits, e.g. 10

You can change the prefix according to your needs.

Step 5: Choose the data set to be transformed

The data set to be transformed should be a backup copy or an archive, delivered by the Model9 agent:

  • <DATA-SET> - the name of the data set

  • <false|true> - if the data set is an Model9 archive, set to “true”. If the data set is a Model9 backup copy, set to “false”.

To change the attributes of the input and the output, and for a full description of the service parameters, see Service parameters.

Step 6: Run the JCL

Submit the job and view the output. See Service response and log samples for sample output.

Step 7: Access the transformed data

Based on the returned response, the outputName will point to the path inside the bucket where the transformed data resides. See Service response and log .

Step-by-step installation

This guide will instruct you how to set up and invoke the transform service of the Model 9 Cloud Data Gateway from the mainframe, using JCL. The service will transform a Model9 data set backup copy / archive into a readable file in the cloud. Once transformed, the readable file can be accessed directly or via data analytics tools.

If using the Model 9 Cloud Data Gateway as an on-premises service, refer to Step-by-step deploymentguide before continuing to the steps on this guide.Step-by-step deployment

Note

The transform service is invoked as SaaS by using the url:

https://us-east-1.model9api.io/

Step 1: Set up
Verify Model 9 Cloud Backup and Recovery for z/OS is installed

Model9 is responsible to deliver the data set from the mainframe to the cloud / on-premises storage. The data set is delivered as a backup copy or an archive, and provides the input to the transform service.

Download z/OS cURL

This free tool will allow you to invoke the transform service from z/OS. If cURL is not installed under /usr/bin, edit line 4 and add the path where the cURL module resides.

Step 2: Copy the script

Copy the following script to /usr/lpp/model9/transformService.sh:

#!/bin/sh
json=$1
url='https://us-east-1.model9api.io/transform'
export PATH=/usr/bin:/bin
export _EDC_ADD_ERRNO2=1
cnvto="$(locale codeset)"
headers="Content-Type:application/json"
echo "Running Model9 transform service"
output=$(curl -H "$headers" -s -k -X POST --data "$json" $url)
if ! [ -z "$output" ]; then
   echo "Transform ended with the following output:"   
   # If the answer is in ASCII then convert to EBCDIC            
   firstChar=$(echo $output | cut -c1)                           
   if [ "$firstChar" = "#" ]; then                               
      convOutput="$(echo $output | iconv -f ISO8859-1 -t $cnvto)"
   else                                                          
      convOutput=$output                                         
   fi                                                            
   echo "$convOutput"                                            
fi
status=$(echo $convOutput | tr -s " " | cut -d, -f1 | cut -d" " -f3)
   echo "Transform ended with status: $status"
if [ "$status" = '"OK"' ];then
   exit 0
else if [ "$status" = '"WARNING"' ]; then
   exit 4
else
   exit 8
fi
fi
Step 3: Copy the JCL

Copy the following JCL to a local library, update the JOBCARD according to your site standards:

//M9TRNSFM JOB 'ACCT#',REGION=0M,CLASS=A,NOTIFY=&SYSUID
//EXTRACT  EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH /usr/lpp/model9/transformService.sh
//         DD *,SYMBOLS=EXECSYS
'{
    "input": {
        "name"   : "<DATA-SET>",
        "complex": "group-&SYSPLEX",
        "archive": "<false|true>"
    },
    "output": {
        "prefix"      : "/transform/&LYR4/&LMON/&LDAY",
        "compression" : "none",
        "format"      : "text"
    },
    "source": {
        "url"     : "<URL>",
        "api"     : "<API>",
        "bucket"  : "<BUCKET>",
        "user"    : "<USER>",
        "password": "<PASSWORD>"
    }
}'
/*
//
Step 4: Customize the JCL
Update the object storage details

Copy the following object storage variables from the Model9 agent configuration file:

  • <URL>

  • <API>

  • <BUCKET>

  • <USER>

  • <PASSWORD>

Update the complex name

The “complex” name represents the group of resources that the Model9 agent can access. By default, this group is named group-<SYSPLEX> and it is shared by all the agents in the same sysplex. The transform JCL specifies the default, using the z/OS system symbol “&SYSPLEX”.

Note

If the default was kept for “complex” in the Model9 agent configuration file, no change is needed

If the “complex” name was changed in the Model9 agent configuration file, change the “complex” in the JCL accordingly.

Update the transform prefix

By default, the JCL will create a transformed copy of your input data set, in the same bucket, with the prefix: /transform/&LYR4/&LMON/&LDAY. The prefix is using the following z/OS system symbols:

  • &LYR4 - The year in 4 digits, e.g. 2019

  • &LMON - The month in 2 digits, e.g. 08

  • &LDAY - The day in the month in 2 digits, e.g. 10

You can change the prefix according to your needs.

Step 5: Choose the data set to be transformed

The data set to be transformed should be a backup copy or an archive, delivered by the Model9 agent:

  • <DATA-SET> - the name of the data set

  • <false|true> - if the data set is an Model9 archive, set to “true”. If the data set is a Model9 backup copy, set to “false”.

To change the attributes of the input and the output, and for a full description of the service parameters, see Service parameters.

Step 6: Run the JCL

Submit the job and view the output. See Service response and log samples for sample output.

Step 7: Access the transformed data

Based on the returned response, the outputName will point to the path inside the bucket where the transformed data resides. See Service response and log .

Transform On-Premises Deployment

This guide will instruct you how to set up the transform service of the Model9 Cloud Data Gateway as an extension to the Model9 Cloud Data Manager server.

Once the service is up and running, see the Step-by-step installation for instructions on how to invoke the service from the mainframe, using JCL.

The service will transform a Model9 data set backup copy / archive into a readable file in the cloud. Once transformed, the readable file can be accessed directly or via data analytics tools.

Note

NOTE: This guide describes how to implement the Model9 Cloud Data Gateway on top of the Model9 Cloud Data Manager installation only.

Step 1: Upload the Model9 Cloud Data Gateway file

The installation file will be provided by your Model9 representative according to your environment. Upload the relevant file to the server using binary mode.

Table 1. The available installation files are:

Environment

Installation File

x86

model9-app-transform_<version>_build_<id>.docker

Linux on Z

model9-app-transform_<version>_build_<id>-s390x.docker



Note

NOTE: <version> represents the version number. <id> represents specific release ID.

Create a work directory under $MODEL9_HOME. The directory should be able to hold at least 20 GB of data:

# Change user to root
sudo su -
# If you haven't done so already, set the model9 target installation path
export MODEL9_HOME=/data/model9
# Change the directory to $MODEL9_HOME
cd $MODEL9_HOME 
mkdir $MODEL9_HOME/extract-work
Step 2: Depoly the Model9 Transform service's component

Deploy the application components using the following commands:

# On Linux issue:
docker load -i $MODEL9_HOME/model9-app-transform_<version>_build_<id>.docker

# On Linux on z issue:
docker load -i $MODEL9_HOME/model9-app-transform_<version>_build_<id>-s390x.docker

Note

NOTE: <version> represents the version number. <id> - Represents specific release ID.

Step 3: Start the service

Start the service using the following command:

docker run -d -p 8443:8443 -v $MODEL9_HOME/extract-work:/data/model9/extract-work:z \
-v $MODEL9_HOME/keys:/data/model9/keys:z \-e "JAVA_OPTS=-Xmx8g" \
-e "SECURE=true" \
-e "WORK_DIRECTORY=/data/model9/extract-work" \
-e "LICENSE_KEY=<license-key>" \
--restart unless-stopped \
--name model9cg-v<version> model9/transform:<version>.<id>

Note

NOTE: <version> represents the version number. <id> represents specific release ID. <license-key> represents the license key obtained from the Model9 representative or support.

Table 2. Supported environment variables:

Environment Variable

Description

Default

JAVA_OPTS

JVM options for the transform service process

None

PORT

The port the service will listen for requests on

5000 / 8443 (when SECURE)

WORK_DIRECTORY

The work directory used by the service

/tmp

SECURE

Whether the service will listen for TLS connections

false

KEYSTORE_FILE_PATH

The file path to the keystore file the service will use when running in secure mode

/data/model9/keys/model9-backup-server.p12

KEYSTORE_PASSWORD

Keystore password

model9

TRUSTSTORE_FILE_PATH

The file path to the truststore file the service will use when running in secure mode

/data/model9/keys/model9-backup-truststore.jks

TRUSTSTORE_PASSWORD

Truststore password

model9

LICENSE_KEY

The Model9 license key

None

PROXY_HOST

Proxy host for the service to use for outgoing connections

None

PROXY_PORT

Proxy port

None

PROXY_EXCLUDE_LIST_FILE_PATH

The path to a file containing URI regular expressions (one per line) which will not be routed via the proxy

None

CHUNK_READAHEAD_BUFFER_SIZE

The size of a single chunk readahead buffer

209715200 (200mb)

CHUNK_READAHEAD_TOTAL_BUFFERS_SIZE

The total size of all chunk readahead buffers

3221225472 (3gb)

CHUNK_READAHEAD_THREAD_POOL_SIZE

Chunk readahead thread pool size

15

COMPRESSION_MANAGER_THREAD_POOL_SIZE

Compression manager thread pool size

15