How to suppress “Entering/Leaving…” messages when invoking make recursively?

Posted on

The “Entering/Leaving…” messages when invoking another make by a make is kind of annoying. Is it possible to suppress these messages and how to suppress them? For GNU make, it is controlled by options: -w, –print-directory Print a message containing the working directory before and after other processing. This may be useful for tracking down
Read more

How to add a new HDFS NameNode metadata directory to an existing cluster?

Posted on

We have a running HDFS cluster. Currently, the NameNode metadata data directory has only one directory configured in hdfs-site.xml: <property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/hdfs/</value> <description>NameNode directory for namespace and transaction logs storage.</description> </property> We would like to add a new directory for dfs.namenode.name.dir to make replicas of the metadata on a separated disk for higher data reliability.
Read more

How to resize a batch of images on Linux?

Posted on

Resizing images on Linux with gThumb is easy. However, I have a batch of images inside a folder. Manually resizing them will consume too much time. How to automatically resize them on Linux with a script? You can use convert from ImageMagick together with bash script to resize images inside a directory. mkdir resize; IFS=$(echo
Read more

Fixing “Remote Host Identification Has Changed” Error When SSH to a Server

Posted on

If you encounter an error message like the one below when attempting to SSH to a server: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has
Read more

How to add a “status bar” to screen on Linux?

Posted on

I noticed that some guys’ screen console has a status bar with tab numbers. That will be very useful for 1) know you are using screen rather than a normal terminal. 2) which tab you are working in. Below is my ~/.screenrc: hardstatus alwayslastline hardstatus string ‘%{= kG}[ %{G}%H %{g}][%= %{=kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B}%Y-%m-%d %{W}%c %{g}]’ It
Read more

How to check the replication factor of a file in HDFS?

Posted on

A related question: how to find the replication factors of files in a HDFS cluster? method 1: You can use the HDFS command line to ls the file. The second column of the output will show the replication factor of the file. For example, $ hdfs dfs -ls /usr/GroupStorage/data1/out.txt -rw-r–r– 3 hadoop zma 11906625598 2014-10-22
Read more

How to download a rtmp stream on Linux?

Posted on

How to download a rtmp video stream on Linux? You can use mplayer to dump the rtmp stream like: mplayer -dumpstream rtmp://example.com/path/to/stream.mp4 It will generate ./stream.dump and you can rename it to the file with the extension you need like stream.mp4. The rtmp link usually can be found from the HTML or JavaScript source code
Read more

How to change an running HDFS cluster’s replication factor?

Posted on

Now, I have a running HDFS cluster storing lost files. I want to change its default replication factor. How to change it? What will happen after it is changed? For example, I change from 2 to 3. Will HDFS automatically re-replicate the data chunks? First, the replication factor is client decided. Second, the replication factor
Read more

how to remove specific directories recursively

Posted on

How to remove .svn directories under hlfs dir recursively as follows. weiwei@weiwei-HP-Compaq-dx6128-MT-PX478AV:~/workshop1/hlfs > find ./ -name “.svn” ./test/build/.svn ./test/.svn ./output/conf/.svn ./output/lib32/.svn ./patches/.svn ./src/include/.svn ./src/include/api/.svn ./src/snapshot/.svn ./src/snapshot/unittest/build/.svn ./src/snapshot/unittest/.svn ./src/utils/.svn ./src/clean/Mapreducer/build/.svn ./src/clean/Mapreducer/.svn ./src/clean/.svn ./src/clean/unittest/.svn ./src/icache/.svn ./src/icache/unittest/.svn ./src/backend/.svn ./src/storage/.svn ./src/cache/.svn ./src/cache/unittest/.svn ./src/clone/.svn ./src/tools/.svn ./src/tools/unittest/.svn ./src/logger/.svn weiwei@weiwei-HP-Compaq-dx6128-MT-PX478AV:~/workshop1/hlfs > find ./ -name “.svn” | xargs rm -rf

How to install gitbook?

Posted on

How to install gitbook on my own Linux box? First, install node.js following https://www.systutorials.com/qa/1268/how-to-install-node-js-on-fedora or How to install node.js on Ubuntu/Linux Mint depending on your distro. Second, install gitbook by npm to /opt/: # cd /opt/ # npm install gitbook Then, the gitbook can be invoked by /opt/node_modules/gitbook/bin/gitbook.js You may need to install the latest
Read more

How to balance DataNode storage in HDFS?

Posted on

As nodes are added and deleted in a Hadoop cluster. Storage usage across DataNodes may be different. Some DataNodes’ disks are almost used up while some others’ are almost empty. How to balance data across DataNodes in HDFS? Hadoop provides the balancer to redistribute the data. Brief introduction to balancer in Hadoop: balancer. The design
Read more

How to find which package can be installed for a file, like “yum provides”?

Posted on

How to find which package can be installed for a file, like “yum provides”? That is, the package is not installed yet and I do not know the package for a file that I want. The apt-file tool can do the similar things as yum provides. You may need to install it first by sudo
Read more

How to run gitbook on a headless server (make Calibre run in headless server)?

Posted on

When use gitbook to generate ebook, Calibre reports this: RuntimeError: X server required. If you are running on a headless machine, use xvfb After xvfb is installed, it does not work either. How to make gitbook/Calibre work on a headless server? You need to wrap the command ebook-convert with xvfb-run. However, in gitbook (lib/generate/ebook/index.js), ebook-convert
Read more

How to config network in host (wireless network) for QEMU guest os

Posted on

The host os is connected into network by wireless one so how to let its guest os connect network. Take [1] as a reference. I run it successfully with following steps. 1, create /etc/qemu-ifup script and chmod it. 2, start a qemu guest os with command sudo ./qemu/qemu-system-x86_64 -enable-kvm -m 1024 -drive file=marss_dram.qcow2 -vnc 127.0.0.1:0
Read more