erve_traits Entity Traits 360 consolidates features and traits of an entity from various entity variables and ML models into a unified view. type: list properties id_served List of all the ID types that will be served. type list
Either complete it or remove it.
as not wanting to use all the input source tables,
in certain cases
ccessing private Git repos from CLI
Using private git repos via CLI
ars Dependency List Note This is applicable to schema 42. With introduction of entity vars as models in schema 44, dependency is done away with. While defining vars in feature_table (entity_var and input_var), it is advisable to list dependencies of each var in the yaml. This helps the system figure out the dependency chain of vars which helps us only create the features that are
Can remove this
ca
Complete 360 degree view Traits defined using entity vars and ML models can all be unified using traits360 models. More information here(link)
For example, from your ID stitching model if main_id is the stitched identifier of a user entity, then each row in feature table will have this as first column and will look like something this:
Add a section on entity var table. This table, as shown below, is a var_table which brings together all features defined. across all vars in var_groups.
D-Stitching table name: This is supposed to be the name of the id stitcher model from the same YAML. The stitched unique ID from this table is used as identifier for the traits of entities of same type in feature t
Users don't need to know this. Only the entity for which they are creating and the input tables that would be used in creating them.
Create a table or view of features from different PB models, using specified ID as a key (such as email, userID, etc). So, a single unified view can have all entity vars, Python models, etc.
Consider that features/traits of an entity are spread across multiple entity vars and ML models. Entity Traits 360 is a means to get them together into a single view. These models are usually defined in pb_project.yaml by creating entries under serve_traits key with corresponding entity. SQL template models can be used as an input to an entity-var/ input-var or as an edge-source in id-stitcher.
es are supported - id_stitcher, feature_table_model and py_sql_model.
This seems old. Instead give a link to section that covers ID types supported.
package lib_ft # app_project imports lib_ft - inputs - tbl_a # Input tables assumed in the imported library package - tbl_b - tbl_c - models - id_stitcher # Id stitcher table from the imported package - feature_table # Feature table defined in the imported package - inputs - tbl_a' # Input tables a' has to be defined in app project with table names as in the warehouse and other details as in package. Then input a has to be remapped to a'. - tbl_b'
Remove this
Feature Table - This allows you to create output tables with computations such as aggregation, average, etc. on selected field
Mark it as legacy. Warn that it is now a view and not a table. Move to the end of the list.
Also not, it does not consider deletes when inferring delta, only updates or inserts.
Remove this line.
The file models/profiles.yaml does not includ
Cover how we specify id stitcher edges. Not how we don't.
Need to add a section on how any column mentioned in ids is automatically sent to the Id Stitcher. Unless, overridden using to_default_stitcher; false
Mention that we do it because Id Stitcher edges can be defined across multipe projects this way.
Also, that this can be done inputs, SQL models, pynative models as well.
model_folders: - models # Name of subfolder inside your project folder, which contains YAML model files # Model folders to use. model_folders: - models
model_folders is duplicated.
on: 2
Should be atleast 5
e is under cons
Remove this. Also, we shouldn't duplicate. We can link to Scheduling scection.
load this file from here and customise these fields: validity_time - Fetch data which was loaded upto this ti
This is still old and incorrect
ributi
Add empty line before comment, for better readability.
ml-features.yaml
Should rename to my-models.yaml
want to create id stitcher and feature table.
Entity has an existence irrespective of whether we want to create it for id stitcher and feature table or not.
Repharse to "Name of the entity"
ame, type and filters (if any) fo
...contains declarations entity objects to be used in ...
string
This is a list of strings.
Timestamp
Model Input Timestamps
ut file ou
Models section should also be simiar.
object
should be list
object
Type is a list of strings.
on of file contains name and id_types. type list
Children not coming.
ion: 2
schema version nicorrect.
inputs.yaml
inputs.yaml before ml-features.yaml
mething like Material_final_id_stitcher_<rest of g
Tables created are as you mentioned. But, there is a view also created with name final_id_stitcher.
ut: It is the name of input table defines in
1) Grammar is incorrect. 2) Edge sources are refs. Usually, they ref to inputs liusted in inputs.yaml. 3) We need a common place to put explanation of refs.
Stitcher
Section can be named just examples.
AML is a dictionary with a list of id stitcher and feature table model:
Need a detailed description on how Id Stitcher uses edge sources to stitch IDs.
resource_type
resource_type is gone. Move description to model_desc.
object
type string
model_type: id_stitcher
Correct the indentation.
object
edge sources is a list of a certain type
ture Table Code Examples¶
Section name too long. Just "Examples" is enough
Other
Why is this a top level section? Have sections which represent the structure.
Regular expression or SQ
Not a regex.
ere you define the source data, along with any conditions.
Expand
Each model YAML is a dictionar
Need better description of what a feature table is.
micro
macro
: blacklistDomains filter: (context_sources_job_run_id = (select top 1 context_sources_job_run_id from blacklistDomains order by timestamp desc)) value: max(case when exclude_reason is not null then 1 else 0 end) - tablevar: name: domainSummary_account_type ref: input: domainSummary value: domainSummary.account_type - feature: name: ignore_domain value: max(case when blacklistFlag = 1 or domainSummary_account_type like '%free%' then 1 else 0 end) description: Whether a domain should be ignored for the analysis - macro: name: call_conversion value: type = 'Call' or tasksubtype='Call' or type like '%demo%') and \ (rlike(lower(subject), '.*\\bcall\\b.*') or rlike(lower(subject), '.*\\bdemo\\b.*' - tablevar:
This is too long. Keep only a representative subset of features.
e Feature table sample.
THis is old schema. Also, we have enough examples in docs repo. Why do we need an external one?
resource_type
resource_type is old term.
micro
macro
string
The type should be ModelRef. We need ModelRef explained at some common place.
In addition to the ones in Identity Stitching, a Feature Table YAML contains the following keys: features
Document schema similar to id stitcher. Additional sections for tabklevar, tablefeature etc can also be put.
stitchier
spell
- type: exclude value: "na" rudder_anon_id: email: filters: - type: include regex: "[A-Za-z0-9+_.-]+@(.+)" inputs: # For every input model, we need to know the column expression which gives identity of each row in the column. - name: websitePageVisits ref: { table: autotrack.pages, timestamp_col: timestamp } ids: - sql: "anonymous_id" type: rudder_anon_id - sql: "case when lower(coalesce(context_traits_email, user_id)) like any ('%gmail%', '%yahoo%') then lower(coalesce(context_traits_email, user_id)) \ else split_part(lower(coalesce(context_traits_email, user_id)),'@',2) end" type: emai
This syntax is also old.
models/id_stitcher.yaml - Download this file from here and customise these fields: validity_time - Fetch data which was loaded upto this timestamp. types - The type of data that can come via ID Stitching. filters - Filter(s) that the type should go through before being included. Filters are processed in order. Current filters enable one to include / exclude specific values or regexes. inputs - List of all input sources along with SQL for fetching them. name - Give a name in which you will store the fetched data. ref - Table in DW you are fetching from, along with time
This is old. Needs to be fixed. We are not downloading files from anywhere now.
Validate
Ensure case correctness.
s under construc
Bring it at parity with Identity Stitching in terms of YAML documentatino.
L object with following keys:
Use JsonSchema to dcument this as well.
odels and Featu
Separate sections for Inputs and Models. Descrien the json-schema of inputs file and models file. Exclude details of modelSpec under models.
for a feature table. We use these id to retrieve values from various tables. Under this we have filters sectio
Can we also cover name and filters as a separate list item?
# Project n
Comment in a separate line, before the target of the comment. Like
. # Project name
. name: sample_attr
TBD
We should get rid of TBD here.
e it in list of project, click on details
...see it in the list of projects. Click on details.
On
Step 1 should be the following. Make tjhe URL clickable. 1) Go to RudderStack web app at app.rudderstack.com.
uld have git repo that contains configurati
You should have already created a git repository from your WHT project.
out this link on ma
Link should be working on the work "this", rather than Git Repo.
Also, Git Repo -> git repository
Requirments
Pre-Requisites
. 8 Folder wise list of art
We should also talk about the ability to click on the log file. NAme of the default log file would also help being mentioned would also help.
After
Is this accurate? After you create a WHT project, you should be able to see it in the list of all WHT projects. From there, if you click on one, you should be able to see all its runs. From the runs, you should be able to click on any one run and see artefacts. Please correct me if I am wrong.
scheduling
Add fullstop at the end.
information
... data warehouse credentials including ...
will
...that will contains.. -> ..that contains..
TBD
We should get rid of TBD here.
o pywht’s
Should call it RS360
the forma
Is this argument supported only in wht run and not compile. Please confirm.
argument
...argument passed. --> The argument passed can be any of feature, macro, tablevar.
here
can be used to
ified breakpoi
Define breakpoint
080_cli_reference
Link not working.
wht run does a check of whether the used role have access to read from all the input tables, write to schema and common in order to store the output of material and other information. If the privileges assigned to the role are suffecient th
1) This section is not specific to any waerhouse. It should be out of the tab. 2) Give a link to wht run documentation in CLI Reference page. 3) Add a line on wht validate access command(with link to CLI reference page) also.
RUDDER_360
RUDDER_360 -> RS360_RUNNER_ROLE
Notes
Add a section in release notes, under version 0.2.0. Label it <red>WIP</red>. Then add a section on access validations
ny user that wants to run the project shoul
"You have to create a new role which is a union of both read and write privileged roles. Any user that wants to run the project shoul...." --> "TLDR; You need to create an additional role specified as a union of the two roles. WHT runs need to be able to read the input tables and write results back to a schema in the warehouse. Furthermore, each run is executed using a single role, specified in the matching profile's section of the site config. It is best in terms of security to create a new role which has read access to all relevant inputs and write access to the output schema. Alternative is to reuse an existing role which has atleast those permissions.
om the input table
You can also add a question on how to test access. Answer with ref to wht validate access command.
We suggest to use
This is not an option. We require creation of atleast one role with both raed and write previliges. Let us discuss.
llowing section describes how you can debug in case you want to troubleshoot the execution of data models.
Can you divide the page into a few sections for better readability. Keep your query under Access Control
permission
permissions
privilege
previleges
have
have -> has
ss
Having a figure here would be quite impactful. Use figure making tool as being used currently.
have their own access control mechanisms, the model(DAC/RBAC) used for access control depends on the warehouse also. Snowflake uses a combination of DAC and R
Can be moved below WHT Projects/Models and Features
See ID stitcher sample. Previous Next
Different releases would end up with different schema versions. Here, we should either say latest version, or have different tags in https://github.com/rudderlabs/RS360-Samples/blob/main/models/id_stitcher_sample.yaml.
hence the results can incorrect. Previous Next
...ID, hence the results can incorrect -> ...ID. If it is ever null the results may be incorrect.
te Configu
Let us move site configuration above WHT Project. Makes more sense and helps us feel a bit bigger.
n tasks on the RudderStack web app.
This is incomplete right now. Anything that is work in progress should be labelled TBD.
<command> <subcommand> [parameters]
In future, please use text instead of image below.
1. resources The key with which yo
Not a great way to describe the resources key.