I need to add delta data to table x. So, I need to create a cycle. I read table x, then I find the delta between the source and table x. After that, I insert the delta to table x. How to do this in dbt? Is it possible to break the cycle?

3 Answers

I change it from ref to source. so instead of using ref, I use source. I create another schema for the source. just to break the cycle in dbt. so basically, 1 table has ref and source at the same time.

what you need is an incremental model, have a look at: incremental models in DBT

6

If the full scan approach is fine, then you can just create a query

select col1,col2, col2... from source_Table minus select col1,col2, col2... from x

this will give you all the records that are there in source_table but not in table x.

If the full scan is not fine, then you can need to make sure that you have updated_timestamp in your source table, and based on that you can write an incremental model.

2

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.