bin/spark-shell -jars $HOME/oci-hdfs/lib/oci-hdfs-full-1.2.7.jar -driver-class-path $HOME/oci-hdfs/lib/oci-hdfs-full-1.2.7.jar We need to reference the JAR file before starting the Spark shell. You receive an error at this point because the oci:// file system schema is not available. With the data ready, we can now launch the Spark shell and test it using a sample command: cd $SPARK_HOME
# Create or copy your API key into the $HOME/.oci directory ForĪdditional information, see HDFS Connector for Object Storage. For production scenarios you would instead put these files in a common place that enforces the appropriate permissions (that is, readable by the user under which Spark and Hive are running).ĭownload the HDFS Connector to the service instanceĪnd add the relevant configuration files by using the following code example. Note For the purposes of this example, place the JAR and key files in the current user's home directory. sbin/start-master.sh Download the HDFS Connector and Create Configuration Files # Should be something like: Scala code runner version 2.12.4 - Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.Įxport SPARK_HOME=$HOME/spark-2.2.1-bin-hadoop2.7 # Should be something like: OpenJDK Runtime Environment (build 1.8.0_161-b14)
#Download spark cassandra connector install#
Sudo yum install java-1.8.0-openjdk.x86_64Įxport JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk # We'll use wget to download some of the artifacts that need to be installed Install Spark and its dependencies, Java and Scala, by using the code examples that follow.Connect to your service instance using an SSH connection.For guidance, see Connecting to an Instance. Ensure that your service instance has a public IP address so that you canĬonnect using a Secure Shell (SSH) connection.Create an instance of your Compute service.
#Download spark cassandra connector archive#
Required third party dependencies are bundled under the third-party/lib folder in the zip archive and should be installed manually. Note Versions 2.7.7.0 and later no longer install all of the required third party dependencies.