Snowflake and Geospatial
I was working on proof of concept to migrate from RDBMS data warehouse to Snowflake data warehouse and one of the requirements we had was to support goespatial during ETL. Snowflake doesn't support geospatial and our requirements were find distance between 2 points find country/state info for a given point find timezone for a given point. No lookup for street addresses was required. Number of records to be processed daily are in millions Input : Records in stage table Table structure Id EventDatetime Lat Lon below is the design and approach I used to meet the above requirements Requirement : Calculate distance and find US state for each record based on lat/lon All record that needs to be processed were loaded into a stage table called eventStage. For POC purpose, table has 53 millions records. Calculate distance To calculate the distance , I used snowflake function HAVERSINE which can calculate the distance between 2 points. Below is...