pyspark.sql.SQLContext.createDataFrame#

SQLContext.createDataFrame(data, schema=None, samplingRatio=None, verifySchema=True)[source]#

Creates a DataFrame from an RDD, a list, a pandas.DataFrame, or a pyarrow.Table.

When schema is a list of column names, the type of each column will be inferred from data.

When schema is None, it will try to infer the schema (column names and types) from data, which should be an RDD of Row, or namedtuple, or dict.

When schema is pyspark.sql.types.DataType or a datatype string it must match the real data, or an exception will be thrown at runtime. If the given schema is not pyspark.sql.types.StructType, it will be wrapped into a pyspark.sql.types.StructType as its only field, and the field name will be “value”, each record will also be wrapped into a tuple, which can be converted to row later.

If schema inference is needed, samplingRatio is used to determine the ratio of rows used for schema inference. The first row will be used if samplingRatio is None.

Added in version 1.3.0.

Changed in version 2.0.0: The schema parameter can be a pyspark.sql.types.DataType or a datatype string after 2.0. If it’s not a pyspark.sql.types.StructType, it will be wrapped into a pyspark.sql.types.StructType and each record will also be wrapped into a tuple.

Changed in version 2.1.0: Added verifySchema.

Changed in version 4.0.0: Added support for pyarrow.Table.

Parameters:

dataRDD or iterable: an RDD of any kind of SQL data representation (Row, tuple, int, boolean, etc.), or list, pandas.DataFrame, or pyarrow.Table.
schemapyspark.sql.types.DataType, str or list, optional: a pyspark.sql.types.DataType or a datatype string or a list of column names, default is None. The data type string format equals to pyspark.sql.types.DataType.simpleString, except that top level struct type can omit the struct<>.
samplingRatiofloat, optional: the sample ratio of rows used for inferring
verifySchemabool, optional: verify data types of every row against schema. Enabled by default.

Returns:

DataFrame

Examples

>>> l = [('Alice', 1)]
>>> sqlContext.createDataFrame(l).collect()
[Row(_1='Alice', _2=1)]
>>> sqlContext.createDataFrame(l, ['name', 'age']).collect()
[Row(name='Alice', age=1)]

>>> d = [{'name': 'Alice', 'age': 1}]
>>> sqlContext.createDataFrame(d).collect()
[Row(age=1, name='Alice')]

>>> rdd = sc.parallelize(l)
>>> sqlContext.createDataFrame(rdd).collect()
[Row(_1='Alice', _2=1)]
>>> df = sqlContext.createDataFrame(rdd, ['name', 'age'])
>>> df.collect()
[Row(name='Alice', age=1)]

>>> from pyspark.sql import Row
>>> Person = Row('name', 'age')
>>> person = rdd.map(lambda r: Person(*r))
>>> df2 = sqlContext.createDataFrame(person)
>>> df2.collect()
[Row(name='Alice', age=1)]

>>> from pyspark.sql.types import *
>>> schema = StructType([
...    StructField("name", StringType(), True),
...    StructField("age", IntegerType(), True)])
>>> df3 = sqlContext.createDataFrame(rdd, schema)
>>> df3.collect()
[Row(name='Alice', age=1)]

>>> sqlContext.createDataFrame(df.toPandas()).collect()
[Row(name='Alice', age=1)]
>>> sqlContext.createDataFrame(pandas.DataFrame([[1, 2]])).collect()
[Row(0=1, 1=2)]

>>> sqlContext.createDataFrame(df.toArrow()).collect()
[Row(name='Alice', age=1)]
>>> table = pyarrow.table({'0': [1], '1': [2]})
>>> sqlContext.createDataFrame(table).collect()
[Row(0=1, 1=2)]

>>> sqlContext.createDataFrame(rdd, "a: string, b: int").collect()
[Row(a='Alice', b=1)]
>>> rdd = rdd.map(lambda row: row[1])
>>> sqlContext.createDataFrame(rdd, "int").collect()
[Row(value=1)]
>>> sqlContext.createDataFrame(rdd, "boolean").collect()
Traceback (most recent call last):
    ...
pyspark.errors.exceptions.captured.PythonException: ...