io

Value Members

1. object docs

   

   Stub object necessary due to https://issues.scala-lang.org/browse/SI-8124

   Stub object necessary due to https://issues.scala-lang.org/browse/SI-8124

   Documentation for ops.core.dataframe.io can be found at software.uncharted.sparkpipe.ops.core.dataframe.io

   Attributes

   protected[this]

   See also

   software.uncharted.sparkpipe.ops.core.dataframe.io

2. def read(path: String, format: String = "parquet", options: Map[String, String] = Map[String, String](), schema: StructType = ...)(sparkSession: SparkSession): DataFrame

   

   Create a DataFrame from an input data source

   Create a DataFrame from an input data source

   path

   A format-specific location String for the source data

   format

   Specifies the input data source format (parquet by default)

   options

   A Map[String, String] of options

   schema

   A StructType schema to apply to the source data

   returns

   a DataFrame createad from the specified source

3. def write(path: String, format: String = "parquet", options: Map[String, String] = Map[String, String]())(input: DataFrame): DataFrame

   

   :: Experimental :: Writes a DataFrame to an output data format/location

   :: Experimental :: Writes a DataFrame to an output data format/location

   path

   A format-specific location String for the source data

   format

   Specifies the output data source format (parquet by default)

   options

   A Map[String, String] of options

   returns

   the input DataFrame, unchanged