Argthtjtr

I have GeoJSON data as structtype like this:

root

 |-- features: array (nullable = true)

 |    |-- element: struct (containsNull = true)

 |    |    |-- geometry: struct (nullable = true)

 |    |    |    |-- coordinates: array (nullable = true)

 |    |    |    |    |-- element: array (containsNull = true)

 |    |    |    |    |    |-- element: array (containsNull = true)

 |    |    |    |    |    |    |-- element: double (containsNull = true)

 |    |    |    |-- type: string (nullable = true)

 |    |    |-- properties: struct (nullable = true)

 |    |    |    |-- auswertezeit: string (nullable = true)

 |    |    |    |-- geschwindigkeit: long (nullable = true)

 |    |    |    |-- strecke_id: long (nullable = true)

 |    |    |    |-- verkehrsstatus: string (nullable = true)

 |    |    |-- type: string (nullable = true)

 |-- type: string (nullable = true)

and i will split data in Columns: strecke_id, auswertezeit, strecke_id, verkehrsstatus, geschwindigkeit and coordinates.
Thank you for your help

Here's a bit simplified example, but this will give you the direction to adjust logic based on your own specs:

import sparkSession.implicits._



val geoDF = sparkSession.read.json("./src/test/resources/geo.json")



val resultDf = geoDF.withColumn("exploaded", functions.explode($"features"))

  .select("exploaded.properties.auswertezeit", "exploaded.properties.geschwindigkeit",

    "exploaded.properties.strecke_id", "exploaded.properties.verkehrsstatus")



resultDf.show()

resultDf.printSchema()

Input data (formatted):

{

    "features": [

        {

            "properties": {

                "auswertezeit": "x",

                "geschwindigkeit": 1,

                "strecke_id": 11,

                "verkehrsstatus": "xx"

            }

        },

        {

            "properties": {

                "auswertezeit": "y",

                "geschwindigkeit": 2,

                "strecke_id": 22,

                "verkehrsstatus": "yy"

            }

        }

    ],

    "type": "xyz"

}

Result:

+------------+---------------+----------+--------------+

|auswertezeit|geschwindigkeit|strecke_id|verkehrsstatus|

+------------+---------------+----------+--------------+

|           x|              1|        11|            xx|

|           y|              2|        22|            yy|

+------------+---------------+----------+--------------+



root

 |-- auswertezeit: string (nullable = true)

 |-- geschwindigkeit: long (nullable = true)

 |-- strecke_id: long (nullable = true)

 |-- verkehrsstatus: string (nullable = true)