Skip to content

Latest commit

 

History

History
326 lines (273 loc) · 11 KB

Tutorial.md

File metadata and controls

326 lines (273 loc) · 11 KB
Raw

NGP IRIS 5 Tutorial

Introduction

IRIS 5 is a complete overhaul of the previous versions of IRIS, mainly in terms of its codebase. The general functionality like download from and upload to the HCP are still here, but might differ from previous versions from what you are used to. This document will hopefully shed some light on what you (the user) can expect and how your workflow with IRIS might change in comparison to previous versions of IRIS.

IRIS 5, like previous versions of IRIS, consists of two main parts: a Python package and an associated Command Line Interface (CLI), which are described below.

CLI

IRIS 5 features a CLI like recent versions of IRIS. However, the new CLI is a bit different compared to before; the general structure of subcommands for the iris command are totally different, but it still has the subcommands you would come to expect. A new command, iris_generate_credentials_file, has also been added. It will generate an empty credentials file that can be filled in with your own NGPr credentials.

The iris command

Typing iris --help yields the following:

Usage: iris [OPTIONS] CREDENTIALS COMMAND [ARGS]...

  NGP Intelligence and Repository Interface Software, IRIS.

  CREDENTIALS refers to the path to the JSON credentials file.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  delete-folder    Delete a folder from an HCP bucket/namespace.
  delete-object    Delete an object from an HCP bucket/namespace.
  download         Download files from an HCP bucket/namespace.
  list-buckets     List the available buckets/namespaces on the HCP.
  list-objects     List the objects in a certain bucket/namespace on the...
  simple-search    Make simple search using substrings in a...
  test-connection  Test the connection to a bucket/namespace.
  upload           Upload files to an HCP bucket/namespace.
  • delete-folder: Deletes a folder on the HCP
  • delete-object: Deletes an object on the HCP
  • download:
    • Downloads a file from a bucket/namespace on the HCP
    • iris path/to/credentials.json download --help: 
      • Usage: iris CREDENTIALS download [OPTIONS] OBJECT BUCKET LOCAL_PATH

  Download files from an HCP bucket/namespace.

  OBJECT is the name of the object to be downloaded.

  BUCKET is the name of the upload destination bucket.

  LOCAL_PATH is the path to where the downloaded objects are to be stored
  locally.

Options:
  --help  Show this message and exit.
  • list-buckets: Lists all buckets that the user is allowed to see
  • list-objects: Lists all objects that the user is allowed to see
  • simple-search: Performs a simple search using a substring in order to find matching objects in a bucket/namespace
  • upload:
    • Uploads either a file or a folder to a bucket/namespace on the HCP
    • iris path/to/credentials.json upload --help: 
      • Usage: iris CREDENTIALS upload [OPTIONS] FILE_OR_FOLDER BUCKET

  Upload files to an HCP bucket/namespace.

  FILE-OR-FOLDER is the path to the file or folder of files to be uploaded.

  BUCKET is the name of the upload destination bucket.

Options:
  --help  Show this message and exit.

Example use cases

The following subsections contain examples of simple use cases for IRIS 5. Of course, correct paths and bucket names should be replaced for your circumstances.

Listing buckets/namespaces
iris path/to/your/credentials.json list-buckets
Downloading a file
iris path/to/your/credentials.json download path/to/your/file/on/the/bucket the_name_of_the_bucket path/on/your/local/machine
Uploading a file
iris path/to/your/credentials.json upload destination/path/on/the/bucket the_name_of_the_bucket path/to/your/file/on/your/local/machine
Searching for a file

By default, the simple-search command is case insensitive:

iris path/to/your/credentials.json simple-search the_name_of_the_bucket your_search_string

This can be changed with the --case_sensitive option:

iris path/to/your/credentials.json simple-search --case_sensitive True the_name_of_the_bucket your_case_sensitive_search_string
Delete a file
iris path/to/your/credentials.json delete-object path/to/your/file/on/the/bucket the_name_of_the_bucket
Delete a folder
iris path/to/your/credentials.json delete-folder path/to/your/folder/on/the/bucket/ the_name_of_the_bucket

The iris_generate_credentials_file command

IRIS 5 comes with a new separate command for generating your NGPr credentials: iris_generate_credentials_file. The idea with this command is to make it easier for anyone to ensure the correct structure of their credentials file. Typing iris_generate_credentials_file --help yields the following:

Usage: iris_generate_credentials_file [OPTIONS]

  Generate blank credentials file for the HCI and HCP.

  WARNING: This file will store sensitive information (such as passwords) in
  plaintext.

Options:
  --path TEXT  Path for where to put the new credentials file
  --name TEXT  Custom name for the credentials file
  --help       Show this message and exit.

Simply running iris_generate_credentials_file will generate a blank credentials file (which is just a JSON file) like the following:

{
  "hcp": {
      "endpoint": "",
      "aws_access_key_id": "",
      "aws_secret_access_key": ""
  },
  "hci": {
      "username": "",
      "password": "",
      "address": "",
      "auth_port": "",
      "api_port": ""
  }
}

Package

The updated codebase for IRIS 5 contains some major changes to use of the package, but should still be familiar. The use cases of IRIS 5 is sill intended to be the same as in previous versions. The difference between IRIS 5 and previous versions is the new syntax and names of classes, methods and functions. Everything in IRIS 5 was inspired by the previous implementations of the boto3 library, which means that most functionality should still exist, but in a different form; methods and functions may have new names, and they might be combined or separated.

The HCPHandler class

IRIS 5 comes with a class for handling connections with the HCP:

HCPHandler(
  self, 
  credentials_path : str, 
  use_ssl : bool = False, 
  proxy_path : str = "", 
  custom_config_path : str = ""
)

Generally, a bucket must be mounted before any methods can be executed. If not, IRIS will throw a NoBucketMounted error. A bucket can be mounted with the following code:

from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

Example use cases

Listing buckets/namespaces

There is no need for mounting a bucket when listing all available buckets. However, credentials are still needed. As such, we can list all buckets with the following:

from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

print(hcp_h.list_buckets())
Downloading a file
from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

hcp_h.download_file("path/to/object/in/bucket", "path/on/local/machine")
Uploading a file
from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

hcp_h.upload_file("path/on/local/machine", "path/to/object/in/bucket")
Uploading a folder
from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

hcp_h.upload_folder("path/on/local/machine/", "path/to/folder/in/bucket/")
Searching for a file
from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

print(hcp_h.search_objects_in_bucket("a search string"))
Delete a file
from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

hcp_h.delete_object("path/to/object/in/bucket/")
Delete a folder
from NGPIris.hcp import HCPHandler

hcp_h = HCPHandler("credentials.json")

hcp_h.mount_bucket("myBucket")

hcp_h.delete_folder("path/to/folder/in/bucket/")

The HCIHandler class

IRIS 5 comes with a class for handling connections with the HCI:

HCIHandler(
  self, 
  credentials_path : str, 
  use_ssl : bool = False
)

A token can be requested to the handler, which in turn allows us to make requests to the HCI:

from NGPIris.hci import HCIHandler

hci_h = HCIHandler("credentials.json")

hci_h.request_token()

Example use cases

List index names
from NGPIris.hci import HCIHandler

hci_h = HCIHandler("credentials.json")
hci_h.request_token()

print(hci_h.list_index_names())
Look up information of an index
from NGPIris.hci import HCIHandler
from pprint import pprint
from json import dumps

hci_h = HCIHandler("credentials.json")
hci_h.request_token()

pprint(
    dumps(
        hci_h.look_up_index("myIndex"), # A dictionary is returned, so we use 
        indent = 4                      # dumps and pprint in order to make the 
    )                                   # output more readable
)
Make queries
from NGPIris.hci import HCIHandler
from pprint import pprint
from json import dumps

hci_h = HCIHandler("credentials.json")
hci_h.request_token()

query = {
    "indexName" : "myIndex",
    "facetRequests" : [
        {
            "fieldName" : "ref"
        },
        {
            "fieldName" : "alt"
        }
    ]
}
pprint(
    dumps(
        hci_h.raw_query(query), # A dictionary is returned, so we use 
        indent = 4              # dumps and pprint in order to make the 
    )                           # output more readable
)