I got this error when I tried to save a file with a space in its name into HDFS: $ hdfs dfs -put -f “/home/u1/testa/test a” “/u1/testa/test a” put: unexpected URISyntaxException while the HDFS seems allow spaces in its file names: https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-common/filesystem/model.html . How to achieve the effect of saving the files with spaces in
Read more
Tag: filesystem
How do I force Linux to unmount a filesystem?
Posted onSome time, Linux fails to unmount a filesystem reporting “device is busy”. I understand that this helps avoid data lost by disallowing unmouting a filesystem when it is being used. But for some situations, I am sure there is something wrong happened or I care not data lost, such as a NFS mounting while the
Read more
Where is the source code for the free command on Linux?
Posted onWhere can I find the source code for the free command on Linux? The source code for the free commands (and many more, like kill, ps, top, sysctl) can be found in procps-ng: https://gitlab.com/procps-ng/procps procps is a set of command line and full-screen utilities that provide information out of the pseudo-filesystem most commonly located at
Read more
mkfs refuses to make filesystem with message “is apparently in use by the system; will not make a filesystem here!”
Posted onI have a disk from another server installed on a new server. However, when I try to make a filesystem on it, mkfs reports # mkfs -t ext4 /dev/sdd1 mke2fs 1.42.8 (20-Jun-2013) /dev/sdd1 is apparently in use by the system; will not make a filesystem here! This is a new disk to the new server.
Read more
How to handle missing blocks and blocks with corrupt replicas in HDFS?
Posted onOne of HDFS cluster’s hdfs dfsadmin -report reports: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 The “Under replicated blocks” can be re-replicated automatically after some time. How to handle the missing blocks and blocks with corrupt replicas in HDFS? Understanding these blocks A block is “with corrupt replicas” in HDFS
Read more
How to allow non-root users on Linux to mount and unmount disks?
Posted onOn a Linux box, we may allow non-root users mount and umount certain disks, such as /dev/sde1. How to allow non-root users on Linux to mount and unmount disks? Please check the tutorial at Controlling Filesystem Mounting on Linux by Playing with /etc/fstab: Allow non-root users to mount and unmount filesystems.
How to avoid mounting failures blocking Linux booting?
Posted onSome entries in /etc/fstab may not not critical for booting Linux or even not available until Linux has booted. How to avoid the failures or unavailability of some mounting entries in /etc/fstab blocking the boot process of Linux? Please check the tutorial at Controlling Filesystem Mounting on Linux by Playing with /etc/fstab: Allow non-root users
Read more
How to find the DataNodes that actually store a file in HDFS?
Posted onA file may be splitted to many chunks and replications stored on many datanodes in HDFS. Now, the question is how to find the DataNodes that actually store a file in HDFS? You may use the dfsadmin -fsck tool from the Hadoop hdfs util. Here is an example: $ hadoop fsck /user/aaa/file.name -files -locations -blocks
Read more
How to import OCaml libraries
Posted onHow to import 3rd party libraries (e.g. not standard libraries) in OCaml? An answer by Gabriel Scherer on how to import Batteries is just great and answers this question with much information. Although it is for Batteries, the method is general. The OCaml compiler (or toplevel, etc.) will find with no additional information only the
Read more
Where are the backup files of my iPhone by iTune and how to get the real files from them?
Posted onI backup my iPhone with iTune to my local computer. Where can I find out the backup files of my iPhone by iTune? And how to get the real files (like songs, notes, etc.) from the backup files? The location of the backup files from iTune It depends on the computer’s operating system: Mac: ~/Library/Application
Read more
Forcing Linux to Unmount a Filesystem Reporting “device is busy”
Posted onLinux may report “device is busy” when we try to umount a filesystem. This behavior is reasonable as it can help us avoid data loss by disallowing unmouting a filesystem when it is being used. But for situations when we are sure there is something wrong happened or we care not data lost such as
Read more
How to Clean RAID Signatures on Linux
Posted onRAID systems such as MegaRAID add signatures to disks to maintain the infomration on these didks. When we simply remove these disks and install them to another server, Linux on the new server may detect these RAID signature infomration and refuses to continue write to the disk. Here is one example that mkfs reports “apparently
Read more
How to Get Available Filesystem Space on Linux: a C Function with a C++ Example
Posted onIt is common for programs to write to files in filesystems on disks. However, what if the disk was almost full when your program writes to the filesystem on a disk? For systems software and mission critical programs, it is a better or must-to-do practice to check the available filesystem space before actually writing to
Read more
Controlling Filesystem Mounting on Linux using /etc/fstab
Posted onControlling the mounting of filesystems is a useful technique for managing Linux systems. The mounting configurations are mostly in the /etc/fstab file. In this post, we will discuss 2 common and useful techniques for controlling the filesystem mounting by playing with the /etc/fstab file: allowing non-root users to mount/unmount filesystems and avoiding mounting failures blocking
Read more
SSD Enabled For DreamHost Shared Hosting: Simple Performance Measurement
Posted onSSD is common for VPS and PaaS virtual machines for higher I/O performance. Now, it is coming to shared hosting too. DreamHost states that “Now with solid state drives (SSDs), our standard web hosting loads pages 200% faster”. We ourselves are happy to know this performance improvement with the price kept the same. Good work,
Read more
Shrinking an Ext4 File System on LVM in Linux
Posted onIn Extending a Mounted Ext4 File System on LVM in Linux, we introduced how to extend a mounted ext4 filesystem on LVM Logical volume by adding a new physical volume. It is also common to shrink an ext4 file system as to spare some disk space. In this post, I will discuss how to shrink
Read more
Hadoop Installation Tutorial (Hadoop 2.x)
Posted onHadoop 2 or YARN is the new version of Hadoop. It adds the yarn resource manager in addition to the HDFS and MapReduce components. Hadoop MapReduce is a programming model and software framework for writing applications, which is an open-source variant of MapReduce designed and implemented by Google initially for processing and generating large data
Read more
Extending Mounted Ext4 File System on LVM in Linux
Posted onLVM is a great tool to manage hard disks on Linux—you can abstract the hard drives away and manage logical volumes from volume groups, you can dynamically add or remove hard drives while the file systems on the logical volumes need not to backed up and recovered, and you may create many snapshots of the
Read more
HP Compaq BIOS Crisis Recovery from a Failed BIOS Flashing Using a USB Flash Disk
Posted onI tried to flash my HP Compaq CQ35-240TX’s BIOS to F.15 but unfortunately failed. The computer turned to white screen. When I restarted (powered off then boot again), this is no response any more: the hard disk and caps lock’s LED keep blinding. I thought “Oh my God! I must send it to HP to
Read more
Setting Up Xen DomU on Fedora Linux (Fedora 12)
Posted onCreating file-backed virtual block device (VBD) for Xen virtual machines and installing Fedora 12 in Xen DomU via internet will be introduced. Note that this tutorial is based on a pretty old OS (Fedora 12). But the method here is still valid while some minor details may need to be changed for latest Xen and
Read more