Btukfyl

I am new to big data. I have developed an spark application using Scala to communicate with hive. It is working fine on Intellij idea. But when I am building an jar file with all dependency I am getting the error

Error: Could not find or load main class

After further debugging I have found that when I am including spark SQL or hive then I am getting the error. It is a maven project. Please have a look on my pom.xml

<modelVersion>4.0.0</modelVersion>



<groupId>spk</groupId>

<artifactId>spk</artifactId>

<version>1.0-SNAPSHOT</version>



<dependencies>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>2.0.2</version>

    </dependency>



    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-sql_2.11</artifactId>

        <version>2.0.2</version>

    </dependency>



    <!--<dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-hive_2.11</artifactId>

        <version>2.0.2</version>

    </dependency>-->



    <!--<dependency>

        <groupId>org.apache.hive</groupId>

        <artifactId>hive-exec</artifactId>

        <version>1.2.2</version>

    </dependency>-->



    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->

    <dependency>

        <groupId>org.apache.kafka</groupId>

        <artifactId>kafka_2.11</artifactId>

        <version>2.0.1</version>

    </dependency>



    <!-- https://mvnrepository.com/artifact/com.101tec/zkclient -->

    <dependency>

        <groupId>com.101tec</groupId>

        <artifactId>zkclient</artifactId>

        <version>0.11</version>

    </dependency>



    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->

    <dependency>

        <groupId>org.apache.kafka</groupId>

        <artifactId>kafka-clients</artifactId>

        <version>0.10.0.0</version>

    </dependency>



</dependencies>



<build>

    <sourceDirectory>src/main/scala</sourceDirectory>

    <plugins>

        <plugin>

            <groupId>org.scala-tools</groupId>

            <artifactId>maven-scala-plugin</artifactId>

            <version>2.11</version>

            <executions>

                <execution>

                    <goals>

                        <goal>compile</goal>

                        <goal>testCompile</goal>

                    </goals>

                    <configuration>

                        <args>

                            <!--<arg>-make:transitive</arg>-->

                            <arg>-dependencyfile</arg>

                            <arg>${project.build.directory}/.scala_dependencies</arg>

                        </args>

                    </configuration>

                </execution>

            </executions>

        </plugin>

        <plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-surefire-plugin</artifactId>

            <version>2.11</version>

            <configuration>

                <useFile>false</useFile>

                <disableXmlReport>true</disableXmlReport>

                <!-- If you have classpath issue like NoDefClassError,... -->

                <!-- useManifestOnlyJar>false</useManifestOnlyJar -->

                <includes>

                    <include>**/*Test.*</include>

                    <include>**/*Suite.*</include>

                </includes>

            </configuration>

        </plugin>

    </plugins>

</build>

Thanks in advance.

Thanks for your help. I have got the answer. There are some problem with Intellij Idea build with all dependency.
I have done the following steps:

1) Added to the pom.xml

<plugin>

            <artifactId>maven-assembly-plugin</artifactId>

            <version>3.1.0</version>

            <configuration>

                <archive>

                    <manifest>

                        <mainClass>spk.kspark</mainClass>

                    </manifest>

                </archive>

                <descriptorRefs>

                    <descriptorRef>jar-with-dependencies</descriptorRef>

                </descriptorRefs>

            </configuration>

        </plugin>

2) Go to the project directory and open Ubuntu terminal

3) mvn install

4) mvn clean compile assembly:single or mvn package assembly:single