redshift_load>: Redshift load operations

redshift_load> operator runs COPY statement to load data from external storage on Redshift.

_export: redshift: host: my-redshift.1234abcd.us-east-1.redshift.amazonaws.com

port: 5439

database: production_db user: app_user ssl: true

strict_transaction: false

+load_from_dynamodb_simple: redshift_load>: schema: myschema table: transactions from: dynamodb://transaction-table readratio: 123

+load_from_s3_with_many_options: redshift_load>: schema: myschema table: access_logs from: s3://my-app-bucket/access_logs/today manifest: true encrypted: true region: us-east-1 csv: "'" delimiter: "$"

json: s3://my-app-bucket/access_logs/jsonpathfile

avro: auto

fixedwidth: host:15,code:3,method:15

gzip: true

bzip2: true

lzop: true

acceptanydate: true acceptinvchars: "&" blanksasnull: true dateformat: yyyy-MM-dd emptyasnull: true encoding: UTF8 escape: false explicit_ids: true fillrecord: true ignoreblanklines: true ignoreheader: 2 null_as: nULl removequotes: false roundec: true timeformat: YYYY-MM-DD HH:MI:SS trimblanks: true truncatecolumns: true comprows: 12 compupdate: ON maxerror: 34

noload: true

statupdate: false role_session_name: federated_user session_duration: 1800

temp_credentials: false

Secrets

When you don't know how to set secrets, please refer to Managing Workflow Secret

• aws.redshift.password: NAME

  Optional user password to use when connecting to the Redshift database.

• aws.redshift_load.access_key_id, aws.redshift.access_key_id, aws.access_key_id

  The AWS Access Key ID to use when accessing data source. This value is used to get temporary security credentials by default. See temp_credentials option for details.

• aws.redshift_load.secret_access_key, aws.redshift.secret_access_key, aws.secret_access_key

  The AWS Secret Access Key to use when accessing data source. This value is used to get temporary security credentials by default. See temp_credentials option for details.

• aws.redshift_load.role_arn, aws.redshift.role_arn, aws.role_arn

  Optional Amazon resource names (ARNs) used to copy data to the Redshift. The role needs AssumeRole role to use this option. Requires temp_credentials to be true. If this option isn't specified, this operator tries to use a federated user

Options

• database: NAME

  Database name.

  Examples:

  database: my_db

• host: NAME

  Hostname or IP address of the database.

  Examples:

  host: db.foobar.com

• port: NUMBER

  Port number to connect to the database. Default: 5439.

  Examples:

  port: 2345

• user: NAME

  User to connect to the database

  Examples:

  user: app_user

• ssl: BOOLEAN

  Enable SSL to connect to the database. Default: false.

  Examples:

  ssl: true

• schema: NAME

  Default schema name. Default: public.

  Examples:

  schema: my_schema

• strict_transaction: BOOLEAN

  Whether this operator uses a strict transaction to prevent generating unexpected duplicated records just in case. Default: true. This operator creates and uses a status table in the database to make an operation idempotent. But if creating a table isn't allowed, this option should be false.

  Examples:

  strict_transaction: false

• status_table_schema: NAME

  Schema name of status table. Default: same as the value of schema option.

  Examples:

  status_table_schema: writable_schema

• status_table: NAME

  Table name prefix of status table. Default: __digdag_status.

  Examples:

  status_table: customized_status_table

• table: NAME

  Table name in Redshift database to be loaded data

  Examples:

  table: access_logs

• from: URI

  Parameter mapped to FROM parameter of Redshift's COPY statement

  Examples:

  from: s3://my-app-bucket/access_logs/today

• manifest: BOOLEAN

  Parameter mapped to MANIFEST parameter of Redshift's COPY statement

  Examples:

  manifest: true

• encrypted: BOOLEAN

  Parameter mapped to ENCRYPTED parameter of Redshift's COPY statement

  Examples:

  encrypted: true

• readratio: NUMBER

  Parameter mapped to READRATIO parameter of Redshift's COPY statement

  Examples:

  readratio: 150

• region: NAME

  Parameter mapped to REGION parameter of Redshift's COPY statement

  Examples:

  region: us-east-1

• csv: CHARACTER

  Parameter mapped to CSV parameter of Redshift's COPY statement. If you want to just use default quote character of CSV parameter, set empty string like csv: ''

  Examples:

  csv: "'"

• delimiter: CHARACTER

  Parameter mapped to DELIMITER parameter of Redshift's COPY statement

  Examples:

  delimiter: "$"

• json: URI

  Parameter mapped to JSON parameter of Redshift's COPY statement

  Examples:

  json: auto

Examples:

  json: s3://my-app-bucket/access_logs/jsonpathfile

• avro: URI

  Parameter mapped to AVRO parameter of Redshift's COPY statement

  Examples:

  avro: auto

  avro: s3://my-app-bucket/access_logs/jsonpathfile

• fixedwidth: CSV

  Parameter mapped to FIXEDWIDTH parameter of Redshift's COPY statement

  Examples:

  fixedwidth: host:15,code:3,method:15

• gzip: BOOLEAN

  Parameter mapped to GZIP parameter of Redshift's COPY statement

  Examples:

  gzip: true

• bzip2: BOOLEAN

  Parameter mapped to BZIP2 parameter of Redshift's COPY statement

  Examples:

  bzip2: true

• lzop: BOOLEAN

  Parameter mapped to LZOP parameter of Redshift's COPY statement

  Examples:

  lzop: true

• acceptanydate: BOOLEAN

  Parameter mapped to ACCEPTANYDATE parameter of Redshift's COPY statement

  Examples:

  acceptanydate: true

• acceptinvchars: CHARACTER

  Parameter mapped to ACCEPTINVCHARS parameter of Redshift's COPY statement

  Examples:

  acceptinvchars: "&"

• blanksasnull: BOOLEAN

  Parameter mapped to BLANKSASNULL parameter of Redshift's COPY statement

  Examples:

  blanksasnull: true

• dateformat: STRING

  Parameter mapped to DATEFORMAT parameter of Redshift's COPY statement

  Examples:

  dateformat: yyyy-MM-dd

• emptyasnull: BOOLEAN

  Parameter mapped to EMPTYASNULL parameter of Redshift's COPY statement

  Examples:

  emptyasnull: true

• encoding: TYPE

  Parameter mapped to ENCODING parameter of Redshift's COPY statement

  Examples:

  encoding: UTF8

• escape: BOOLEAN

  Parameter mapped to ESCAPE parameter of Redshift's COPY statement

  Examples:

  escape: false

• explicit_ids: BOOLEAN

  Parameter mapped to EXPLICIT_IDS parameter of Redshift's COPY statement

  Examples:

  explicit_ids: true

• fillrecord: BOOLEAN

  Parameter mapped to FILLRECORD parameter of Redshift's COPY statement

  Examples:

  fillrecord: true

• ignoreblanklines: BOOLEAN

  Parameter mapped to IGNOREBLANKLINES parameter of Redshift's COPY statement

  Examples:

  ignoreblanklines: true

• ignoreheader: NUMBER

  Parameter mapped to IGNOREHEADER parameter of Redshift's COPY statement

  Examples:

  ignoreheader: 2

• null_as: STRING

  Parameter mapped to NULL AS parameter of Redshift's COPY statement

  Examples:

  null_as: nULl

• removequotes: BOOLEAN

  Parameter mapped to REMOVEQUOTES parameter of Redshift's COPY statement

  Examples:

  removequotes: false

• roundec: BOOLEAN

  Parameter mapped to ROUNDEC parameter of Redshift's COPY statement

  Examples:

  roundec: true

• timeformat: STRING

  Parameter mapped to TIMEFORMAT parameter of Redshift's COPY statement

  Examples:

  timeformat: YYYY-MM-DD HH:MI:SS

• trimblanks: BOOLEAN

  Parameter mapped to TRIMBLANKS parameter of Redshift's COPY statement

  Examples:

  trimblanks: true

• truncatecolumns: BOOLEAN

  Parameter mapped to TRUNCATECOLUMNS parameter of Redshift's COPY statement

  Examples:

  truncatecolumns: true

• comprows: NUMBER

  Parameter mapped to COMPROWS parameter of Redshift's COPY statement

  Examples:

  comprows: 12

• compupdate: TYPE

  Parameter mapped to COMPUPDATE parameter of Redshift's COPY statement

  Examples:

  compupdate: ON

• maxerror: NUMBER

  Parameter mapped to MAXERROR parameter of Redshift's COPY statement

  Examples:

  maxerror: 34

• noload: BOOLEAN

  Parameter mapped to NOLOAD parameter of Redshift's COPY statement

  Examples:

  noload: true

• statupdate: TYPE

  Parameter mapped to STATUPDATE parameter of Redshift's COPY statement

  Examples:

  statupdate: off

• temp_credentials: BOOLEAN

  Whether this operator uses temporary security credentials. Default: true. This operator tries to use temporary security credentials as follows:

  • If role_arn is specified, it calls AssumeRole action
  • If not, it calls GetFederationToken action

  See details about AssumeRole and GetFederationToken in the documents of AWS Security Token Service.

  So either of AssumeRole or GetFederationToken action is called to use temporary security credentials by default for secure operation. But if this option is disabled, this operator uses credentials as-is set in the secrets insread of temporary security credentials.

  Examples:

  temp_credentials: false

• session_duration INTEGER

  Session duration of temporary security credentials. Default: 3 hour. This option isn't used when disabling temp_credentials

  Examples:

  session_duration: 1800