Type “ ” as the search string to search within the Maven Central repository. Click on the Libraries and then select the Maven as the Library source. Once you set up the cluster, next add the spark 3 connector library from the Maven repository. You can find more information on how to create an Azure Databricks cluster from here. We used a two-node cluster with the Databricks runtime 8.1 (which includes Apache Spark 3.1.1 and Scala 2.12). In this example, we use a dataset found in the Databricks workspace at d bfs:/databricks-datasets/COVID/coronavirusdataset/SeoulFloating.csv. Tip: You can display the datasets in the workspace by executing the below in a notebook: display(dbutils.fs.ls(“/databricks-datasets”)). In this example, we read from a dataset stored in an Azure Databricks workspace and store it in an Azure Cosmos DB container using a Spark job. (Currently, the Spark 3 OLTP connector for Azure Cosmos DB only supports Azure Cosmos DB Core (SQL) API, so we will demonstrate it with this API) How to ingest data into the Azure Cosmos DB.The differences between the partitioning strategies used.How to use the Catalog API to create a database and a collection in the Azure Cosmos DB account.How to add the spark 3 connector library to an Azure Databricks cluster.In this Spark article, you have learned how to combine two or more DataFrame’s of the same schema into single DataFrame using Union method and learned the difference between the union() and unionAll() functions.This blog post demonstrates how you can use Spark 3 OLTP connector for Azure Cosmos DB (now in general availability) with Azure Databricks to ingest and read the data. This complete example is also available at the GitHub project. Val df2 = simpleData2.toDF('employee_name','department','state','salary','age','bonus') Val spark: SparkSession = SparkSession.builder() As you see, this returns only distinct rows. Since the union() method returns all rows without distinct records, we will use the distinct() function to return just one record when duplicate exists. Now, let’s create a second Dataframe with the new records and some records from the above Dataframe but with the same schema.ĭataFrame unionAll() method is deprecated since Spark “2.0.0” version and recommends using the union() method. |employee_name|department|state|salary|age|bonus| |- employee_name: string (nullable = true) This yields the below schema and DataFrame output. Val df = simpleData.toDF('employee_name','department','state','salary','age','bonus') First, let’s create two DataFrame with the same schema.

Spark 3 vs spark 2