Search for a command to run...
Articles tagged with #spark
Objective: With the orc file created from the Delta table in the post, now we will read that orc file and upsert the Delta table score (which has the same schema as the student table mentioned in that post). Initial setup Create the database %sql c...
Databricks Pyspark to create a ORC file Objective: We will read the data from Delta table and create the IRC file using Spark readStream and writeStream. For this blog I used the Databricks Community edition (free). This approach can also exte...
1. Creating Dataframe programmatically with data and schema for the data from pyspark.sql.types import StructType,StructField, StringType, IntegerType dataValues = [("Tim","100","Sales",4050), ("Don","102","Accounts",5500), ("Rob","103","Secu...