Posts

Installing Higher Python Versions on AWS Cloud9 and Creating Lambda Layers for Snowflake Connector

This blog post addresses two critical aspects for developers working with AWS Cloud9 and Lambda functions: 1. Installing Higher Python Versions on Cloud9 This section guides you through the process of installing a specific version of Python (3. 9 in this example) on your Cloud9 environment, ensuring compatibility with dependencies like urrlib3 >= 2. 0. 0. 2. Creating Lambda Layers for Snowflake Connector with Different Python Versions Learn how to build a Lambda Layer containing the Snowflake connector, enabling its use in your Lambda functions regardless of the default Python version on your Cloud9 environment. Installing Python 3.9 on Cloud9 Preparation: Create a New Cloud9 Environment: Start with a fresh environment to avoid conflicts with existing configurations. Update System Dependencies: Ensure compatibility with Python 3. 9 by running the following commands: sudo yum remove openssl-devel sudo yum install gcc openssl11-devel bzip2-devel libffi-devel Installati...
1 comment
Read more

Snowflake and Geospatial

I was working on proof of concept to migrate from RDBMS data warehouse to Snowflake data warehouse and one of the requirements we had was to support goespatial during ETL. Snowflake doesn't support geospatial and our requirements were find distance between 2 points find country/state info for a given point find timezone for a given point. No lookup for street addresses was required. Number of records to be processed daily are in millions Input  : Records in stage table Table structure Id EventDatetime Lat Lon below is the design and approach I used to meet the above requirements Requirement :  Calculate distance and find US state for each record based on lat/lon All record that needs to be processed were loaded into a stage table called eventStage. For POC purpose, table has 53 millions records. Calculate distance To calculate the distance , I used snowflake function  HAVERSINE which can calculate the distance between 2 points. Below is...
3 comments
Read more