Errors and Fixes

1) What do the four V’s of Big Data denote? a) Volume –Scale of data b) Velocity –Analysis of streaming data c) Variety – Different forms of data d) Veracity –Uncertainty of data 2) On what concept the Hadoop framework works? - Hadoop Framework works on the following two core components 1) HDFS:(Storage Unit) - Hadoop Distributed File System is the java based file system for scalable and reliable storage of large datasets. - Data in HDFS is stored in the form of blocks and it operates on the Master Slave Architecture. 2) MapReduce:(Processing  and Cluster Management Unit) - This is a java based programming paradigm of Hadoop framework that provides scalability across various Hadoop clusters. - MapReduce distributes the workload into various tasks that can run in parallel. Hadoop jobs perform 2 separate tasks- job. - MAP: -The map job breaks down the data sets into key-value pairs or tuples. - Reduce: - The reduce job then takes the output of th...

1) 401:Authentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect.    Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync. Fix: Check for the 3 mentioned details in your Conf file and system clock and try to relaunch again 2) 17/05/05 12:19:54 ERROR hdfs.HDFSEventSink: process failed java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$RenewLeaseRequestProto overrides final method getUnknownFields. ()Lcom/google/protobuf/UnknownFieldSet; Fix: 1) check for your HDFS sink path in the flume conf file and fixx it accordingly.     2) check whether the the HDFS is in Safe mode and retry to leave from it. 3)  ERROR node.PollingPropertiesFileConfigurationProvider: Unhandled error Fix: Download and copy the flume-twitter-dource-1.6.0.jar into lib folder. 2) TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true 4) java.lan...

1) ERROR tool.ImportTool: Error during import: HBase jars are not present in classpath, cannot import to HBase! Fix: import the following into sqoop-env.sh in config folder. 1) export HIVE_HOME=/home/mj/hive-0.13.1 2) export HBASE_HOME=/home/mj/hbase-0.98.8

1) FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict Fix: a) set hive.exec.dynamic.partition.mode=nonstrict; b) set hive.exec.dynamic.partition=true; 2) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. Fix: deploy latest mysql-jdbs jar 3) 20)for ClassNotFoundException: org.apache.hadoop.hive.jdbc.HiveDriver add hive-jdbc JAR 4) java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException add hive-metastore jar 5) java.lang.NoClassDefFoundError: org/apache/thrift/TBase add libthrift JAR 6) java.lang.NoClassDefFoundError: org/apache/hadoop/hive/service/HiveInterface add hive-service Jar 7) java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface add libfb Jar 8) java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory ...

1) hbase shell slf4j class path contains multiple slf4j bindings cause: both Hadoop 2.x and Hbase 0.98 having the duplicated jars. Fix: Remove the following file form hbase/lib/slf4j-log4j12-1.6.4.jar

1) java.lang.NoClassDefFoundError: com/mongodb/connection/BufferProvider Fix: Need to add the following 3 jar's in class path of ur eclipse project - mongodb-driver-x.x.x.jar - mongodb-driver-core-x.x.x.jar (this is the main reason for this error) - bson-x.x.x.jar