pyspark.sql.functions.to_variant_object#

pyspark.sql.functions.to_variant_object(col)[source]#

Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs. Input maps can only have string keys.

Added in version 4.0.0.

Parameters:

colColumn or str: a column with a nested schema or column name A column that evaluates to an array, map, or struct.

Returns:

Column: a new column of VariantType. Returns a column that evaluates to a variant.

Examples

Example 1: Converting an array containing a nested struct into a variant

>>> from pyspark.sql import functions as sf
>>> from pyspark.sql.types import ArrayType, StructType, StructField, StringType, MapType
>>> schema = StructType([
...     StructField("i", StringType(), True),
...     StructField("v", ArrayType(StructType([
...         StructField("a", MapType(StringType(), StringType()), True)
...     ]), True))
... ])
>>> data = [("1", [{"a": {"b": 2}}])]
>>> df = spark.createDataFrame(data, schema)
>>> df.select(sf.to_variant_object(df.v))
DataFrame[to_variant_object(v): variant]
>>> df.select(sf.to_variant_object(df.v)).show(truncate=False)
+--------------------+
|to_variant_object(v)|
+--------------------+
|[{"a":{"b":"2"}}]   |
+--------------------+