{"id":312,"date":"2017-11-17T11:34:26","date_gmt":"2017-11-17T11:34:26","guid":{"rendered":"http:\/\/marcel-jan.eu\/datablog\/?p=312"},"modified":"2017-11-17T11:34:49","modified_gmt":"2017-11-17T11:34:49","slug":"recovering-your-hdp-2-6-1-sandbox-on-virtualbox-after-a-restart","status":"publish","type":"post","link":"https:\/\/marcel-jan.eu\/datablog\/2017\/11\/17\/recovering-your-hdp-2-6-1-sandbox-on-virtualbox-after-a-restart\/","title":{"rendered":"Recovering your HDP 2.6.1 Sandbox on VirtualBox after a restart"},"content":{"rendered":"

If you’ve worked with the Hortonworks Data Platform 2.x sandbox of later versions in VirtualBox and made it shutdown rather vigorously, you might have noticed that you won’t get past this startup screen when you try to start it up the next time:<\/p>\n

\"\"<\/p>\n

I had this a couple of times and that’s why I decided to pause my sandbox every time and save it before shutting down my laptop. But yesterday Windows 10 decided to step in. After a day of studying it was high time for me to have dinner, during which I kept the laptop on. Little did I know that Windows 10 at that time decided to update and restart. And to do this, it needed to shutdown every application. Including VirtualBox. When I came back I found out to my horror that my carefully prepared HDP sandbox was shutdown in the roughest of ways. Thanks, Microsoft!<\/p>\n

I started the HDP 2.6.1 sandbox again, but.. that startup screen never went further. I had the option to start again fresh by importing the VirtualBox anew, transfering my files again, creating my Hive databases again… No! I decided to put a line in the sand. This HDP sandbox will live! “Don’t go into the light.. er.. Horty!”<\/p>\n

\"\"<\/p>\n

Let’s ask Google<\/h2>\n

When you google “hdp sandbox hangs” or “hdp sandbox won’t restart” you almost certainly arrive at the Hortonworks Community. There is some advise there that means you have to start again on a different platform (No!) and some tips to diagnose the situation. For example, you can see how far CentOS got during the startup by pressing an cursor key.<\/p>\n

\"\"<\/p>\n

Now at least you have a better way to describe how far the startup got.<\/p>\n

Even more valuable was this advise<\/a> from Hortonworks Community user Tiago Rubio that you can still access the sandbox in this situation.<\/p>\n

Let’s access Ambari<\/h2>\n

Does this mean I can access Ambari? Yes, yes you can. Give the startup some time, then go to http:\/\/sandbox.hortonworks.com:8080<\/a> and sure enough – well I at least got to see this screen:<\/p>\n

\"\"<\/p>\n

But the news is not all good. Almost nothing is up.<\/p>\n

\"\"<\/p>\n

Start all up<\/h2>\n

So my first intuition was to use the Actions -> Start All button.<\/p>\n

\"\"<\/p>\n

Sure enough Ambari sets to work. Things will work out after all, you’ll see. That is until… you get the results back:<\/p>\n

\"\"<\/p>\n

Click on the\u00a0\"\" a couple of times and you get a list of what components started and which ones didn’t.<\/p>\n

\"\"<\/p>\n

Notice that the startup of the (YARN) Resource Manager was aborted, as was the Node Manager and many other tasks. These are very important.<\/p>\n

Is there a Hadoop engineer in the house?<\/h2>\n

It seems we have to start some stuff by hand. For this you need to know in what order to start things. First we go to the Dashboard and look at HDFS.<\/p>\n

\"\"<\/p>\n

Ah, see Hadoop runs on HDFS. And here the NameNode and the DataNode of HDFS are stopped. Let’s click on the NameNode link. We now get a list of components on the sandbox.hortonworks.com server, which is the only server in this “cluster”.<\/p>\n

\"\"<\/p>\n

Now let’s try to start the NameNode individually.<\/p>\n

\"\"<\/p>\n

In my case it turned out it went stuck at 35% progress for quite a while. What was happening? A couple of clicks on \"\" and I found this message:<\/p>\n

2017-11-17 10:15:24,244 - Retrying after 10 seconds. Reason: Execution of '\/usr\/hdp\/current\/hadoop-hdfs-namenode\/bin\/hdfs dfsadmin -fs \r\nhdfs:\/\/sandbox.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.<\/pre>\n
It took multiple minutes, but eventually the NameNode did start without my interference. You need a bit of patience here. Now on to the next component: the DataNode. You’ll find it a bit lower in the list. This one started pretty quickly.<\/p>\n
We could do everything by hand from here on, but actually at this point I gave that Start All button a try again. And now 100% of the services start.<\/p>\n
\"\"<\/p>\n
Going back to the Dashboard in Ambari I see only green marks:<\/p>\n
\"\"<\/p>\n
Testing it<\/h2>\n
But does it really work? I want proof that what I was working on still works.<\/p>\n
Here is Hive:<\/p>\n
[root@sandbox ~]# su - spark\r\n[spark@sandbox ~]$ export SPARK_MAJOR_VERSION=2\r\n[spark@sandbox ~]$ export HIVE_HOME=\/usr\/hdp\/2.6.1.0-129\/hive\r\n[spark@sandbox ~]$ hive\r\nlog4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.\r\n\r\nLogging initialized using configuration in file:\/etc\/hive\/2.6.1.0-129\/0\/hive-log4j.properties\r\nhive> show databases;\r\nOK\r\nasteroids\r\ndefault\r\nfoodmart\r\nhdpcd_retail_db_orc\r\nhdpcd_retail_db_txt\r\nxademo\r\nTime taken: 2.137 seconds, Fetched: 6 row(s)\r\nhive> use asteroids;\r\nOK\r\nTime taken: 0.291 seconds\r\nhive> select count(*) from asteroids;\r\nOK\r\n745470\r\nTime taken: 0.662 seconds, Fetched: 1 row(s)<\/pre>\n
That is promissing. But let’s see Spark next:<\/p>\n
[spark@sandbox ~]$ pyspark --master yarn-client --conf spark.ui.port=12562 --executor-memory 1G --num-executors 1\r\nSPARK_MAJOR_VERSION is set to 2, using Spark2\r\nPython 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)\r\n[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\nWarning: Master yarn-client is deprecated since 2.0. Please use master \"yarn\" with specified deploy mode instead.\r\nSetting default log level to \"WARN\".\r\nTo adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\r\n\/usr\/hdp\/current\/spark2-client\/python\/pyspark\/context.py:207: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0\r\n warnings.warn(\"Support for Python 2.6 is deprecated as of Spark 2.0.0\")\r\nWelcome to\r\n ____ __\r\n \/ __\/__ ___ _____\/ \/__\r\n _\\ \\\/ _ \\\/ _ `\/ __\/ '_\/\r\n \/__ \/ .__\/\\_,_\/_\/ \/_\/\\_\\ version 2.1.1.2.6.1.0-129\r\n \/_\/\r\n\r\nUsing Python version 2.6.6 (r266:84292, Aug 18 2016 15:13:37)\r\nSparkSession available as 'spark'.\r\n>>> orders = sc.textFile(\"\/user\/dmaster\/retail_db\/orders\")\r\n>>> orders.count()\r\n68883<\/pre>\n
So that all works great. I can do my experimenting again. Phew!<\/p>\n
Are we really done?<\/h2>\n
One more thing. So the Ambari Dashboard looks apple green, but when you look at the components on the sandbox.hortonworks.com server it actually does not.<\/p>\n
\"\"<\/p>\n
I haven’t checked out what status these components had before I brought my HDP sandbox back from the death. I know HDFS’ SNameNode won’t work on only one server, but some others might be necessary for what you are working on. I didn’t went as far as to test this, but you could try starting these components individually and if they don’t start, take a good look at the errors they produce.<\/p>\n
Horty lives<\/h2>\n
So in the end my HDP sandbox is alive again, but the VirtualBox will stay in it’s startup screen. Consider this a bypass operation. Hopefully this helps you to save your work too.<\/p>\n","protected":false},"excerpt":{"rendered":"
If you’ve worked with the Hortonworks Data Platform 2.x sandbox of later versions in VirtualBox and made it shutdown rather vigorously, you might have noticed that you won’t get past this startup screen when you try to start it up the next time: I had this a couple of times […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35,55],"tags":[86,61,62,42,5,23,87,60,84,88],"class_list":["post-312","post","type-post","status-publish","format-standard","hentry","category-apache-products-for-outsiders","category-howto","tag-ambari","tag-datanode","tag-hdfs","tag-hdp-sandbox","tag-hive","tag-hortonworks","tag-horty","tag-namenode","tag-spark","tag-virtualbox"],"_links":{"self":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/312","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/comments?post=312"}],"version-history":[{"count":6,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/312\/revisions"}],"predecessor-version":[{"id":332,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/posts\/312\/revisions\/332"}],"wp:attachment":[{"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/media?parent=312"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/categories?post=312"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marcel-jan.eu\/datablog\/wp-json\/wp\/v2\/tags?post=312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}