Skip to content

Latest commit

 

History

History
65 lines (58 loc) · 3.11 KB

windows-instructions.md

File metadata and controls

65 lines (58 loc) · 3.11 KB

Getting Started with Spark .NET on Windows

These instructions will show you how to run a .NET for Apache Spark app using .NET Core on Windows.

Pre-requisites

For detailed instructions, you can see Building .NET for Apache Spark from Source on Windows.

Authoring a .NET for Apache Spark App

  • Open Visual Studio -> Create New Project -> Console App (.NET Core) -> Name: HelloSpark
  • Install Microsoft.Spark Nuget package into the solution from the spark nuget.org feed - see Ways to install Nuget Package
  • Write the following code into Program.cs:
    using Microsoft.Spark.Sql;
    
    namespace HelloSpark
    {
        class Program
        {
            static void Main(string[] args)
            {
                var spark = SparkSession.Builder().GetOrCreate();
                var df = spark.Read().Json("people.json");
                df.Show();
            }
        }
    }
  • Build the solution

Running your .NET for Apache Spark App

  • Open your terminal and navigate into your app folder:
    cd <your-app-output-directory>
    
  • Create people.json with the following content:
    {"name":"Michael"}
    {"name":"Andy", "age":30}
    {"name":"Justin", "age":19}
  • Run your app
    spark-submit `
    --class org.apache.spark.deploy.dotnet.DotnetRunner `
    --master local `
    microsoft-spark-<version>.jar `
    dotnet HelloSpark.dll
    
    Note: This command assumes you have downloaded Apache Spark and added it to your PATH environment variable to be able to use spark-submit, otherwise, you would have to use the full path (e.g., c:\bin\apache-spark\bin\spark-submit). For detailed instructions, you can see Building .NET for Apache Spark from Source on Windows.
  • The output of the application should look similar to the output below:
    +----+-------+
    | age|   name|
    +----+-------+
    |null|Michael|
    |  30|   Andy|
    |  19| Justin|
    +----+-------+