A template string is a template expression that generates a string such as the path of resource
You can refer to this pattern:
A target pattern is a string that contains at minimal a variable expression.
Variable Expression:
${prefix_variable_name}
# or
$prefix_variable_name
where prefix describe the type of variable provider
| Prefix | Type | Type of variables |
|---|---|---|
| record or rec | record | record_column_values |
| input or in | input data path | data_path_attributes |
| target or tar | target data path | data_path_attributes |
| pipeline or pipe | pipeline | pipeline_attributes |
Each step will describe the accepted variables providers
In the pattern, you can reference a globbing matched wildcard from a data source selector using the $x syntax where x indicates the index of a glob pattern. ie
You can see the value of the back reference with the tabul data list command. If you want to know the value of $1, just ask for the attribute 1
Example with tpcds where we want to capture the name of the web table but without the prefix.
tabul data list -a 1 web_*@tpcds
path media_type 1
----------- ------------ -------
web_page sql/relation page
web_returns sql/relation returns
web_sales sql/relation sales
web_site sql/relation site
In the template, you can reference any attribute of a data path
Example for a logical name of input data resource
${input_logical_name}
You can see all available attributes with the data info command.
tabul data info web_page@tpcds
Information about the data resource (web_page@tpcds)
attribute value description
---------------- ------------------------------------------------------------------------------------------------ -----------------------------------------------------
ABSOLUTE_PATH web_page The absolute path on the data system
ACCESS_TIME The access time (access time)
COMMENT A comment
CONNECTION tpcds The connection name
COUNT The number of records
CREATION_TIME The creation time (birth time)
DATA_URI web_page@tpcds The data uri
KIND relation The kind of media
LOGICAL_NAME web_page The logical name
MD5 2d3f08a0ad007e5236ed694250e89721 The Md5 hash
MEDIA_SUBTYPE relation The media subType
MEDIA_TYPE sql/relation The media type
NAME web_page The name of the data resource
PARENT The parent
PATH web_page The relative path to the default connection path
SHA384 0d3fcd0bd2dc1aa6c14a6ce4a4c8acff06a6194a4f56f4730bbb371aaf490da1fe318f9f11ba44157ffd53d501e2f551 The Sha384 hash
SHA384_INTEGRITY sha384-DT/NC9LcGqbBSmzkpMis/wamGUpPVvRzC7s3Gq9JDaH+MY+fEbpEFX/9U9UB4vVR The sha384 value used in the html integrity attribute
SIZE The number of byte
TABULAR_TYPE data The tabular type
UPDATE_TIME The last update time (modify time)
If your template is used in a pipeline, you can add any pipeline attributes.
Example:
${pipeline_start_time}
If your step is running on record level (Example: split) you can also add the record value of a column using the $rec_x or ${rec_x} syntax where x should be replaced by the column name.
Example:
${record_column_name}
${rec_column_name}