PARSEC is the most important CPU-bound benchmark for systems. It is huge and hard to install because it needs lots of 3-part libs. PARSEC download link for 3.0 version: http://parsec.cs.princeton.edu/download.htm#parsec I remembered I added the answer yesterday night but I could not see the answer currently. Anyway, let me add the answer again after I
Read more
Tag: systems
How to configure systemd to boot Linux to console mode (runlevel 3)?
Posted onHow to configure Linux (am using Fedora 21) managed by systemd to boot to console (init 3) mode? systemd has the concept of targets as a more flexible replacement for runlevels in sysvinit. Runlevel 3 is emulated by multi-user.target. runlevel3.target is a symbolic link to multi-user.target. You can switch to ‘runlevel 3’ by running #
Read more
How to handle missing blocks and blocks with corrupt replicas in HDFS?
Posted onOne of HDFS cluster’s hdfs dfsadmin -report reports: Under replicated blocks: 139016 Blocks with corrupt replicas: 9 Missing blocks: 0 The “Under replicated blocks” can be re-replicated automatically after some time. How to handle the missing blocks and blocks with corrupt replicas in HDFS? Understanding these blocks A block is “with corrupt replicas” in HDFS
Read more
HDFS stays in safe mode because of reported blocks not reaching 0.9990 of total blocks
Posted onAfter a node failure and restarting the HDFS, the NameNode reports: “The reported blocks 1968810 needs additional 5071 blocks to reach the threshold 0.9990 of total blocks 1975856. Safe mode will be turned off automatically.” in the log. Why this happens? And how to fix it? About why the NameNode stays in the safe mode:
Read more
Any good Web browser in command line in Linux?
Posted onWhat suggested web browser in command line? This will be very useful for browsing the Internet from a command line via SSH. You may use Lynx, a fully-featured World Wide Web (WWW) client for users running cursor-addressable, character-cell display devices, in Linux. Homepage: http://lynx.isc.org/ Lynx is a fully-featured World Wide Web (WWW) client for users
Read more
how to skip mapper function in hadoop
Posted onIn hadoop I need to skip mapper function and directly execute the reducer function. We doing this to improve hadoop performance, if the hadoop framework is used to analyze same data sets, then mapper’s output will be same for different kind of jobs. To save the redundant computation for same results, I am planning to
Read more
How to make dd faster on Linux?
Posted ondd seems slow when I use command like # dd if=/dev/sda2 of=./sda2.bak How to make it faster? You can make dd faster by specifying a good bs like # dd if=/dev/sda2 of=./sda2.bak bs=8192 8192 is a magic number. There are may be other good sizes for bs for different systems. But 8192 works pretty well
Read more
How to understand some key system consistency algorithoms
Posted onWhen we design a system, we may want our systems to be consistency, scalability and so on. Currently, there are some famous consistency algorithms. How to understand them easily. 1, Paxos and its extensions 2, Replicated State Machine mechanisms 3, Quorum Welcome to adding other famous consistency algorithms and its understanding ;-) Reading text books
Read more
What are the differences between database DDL and DML
Posted onDifferences beween DDL (Data Definition Language) and DML (Data Manipulation Language) Data Definition Language (DDL) statements are used to define the database structure or schema. Data Manipulation Language (DML) statements are used for managing data within schema objects. References:http://www.orafaq.com/faq/what_are_the_difference_between_ddl_dml_and_dcl_commands
What are the DDL and DML of Shark (Spark SQL)?
Posted onCurrently, I wanna take Shark’s (Spark SQL) DDL and DML as an reference to design/implement SQLE’s DDL and DML. However, I cannot find its DDL and DML. I can only find several SQLs in Shark paper[1]. [1] shark paper – http://tab.d-thinker.org/showthread.php?tid=2585 Shark’s language is Hive QL. HQL’s DDL and DML can be found at Hive
Read more
What is database schema?
Posted onAs is known, database schema is a collection of database rules like data type and data relationship. However, this is still a little abstract. So what is database schema in one sentence everyone could understand it clearly. The wikipedia page on Database schema is quite good for understanding the concept. A database schema of a
Read more
List Files from a .deb Package in Ubuntu Linux
Posted onHow to list all files from a .deb package like rpm -ql on RPM based systems? Use this command dpkg -L pkg_name -L lists files: -L, –listfiles package-name… List files installed to your system from package-name. More on the dpkg command, check dpkg manual.
What is the design of Snapshots in HDFS?
Posted onWhat is the design of Snapshots in HDFS? This PDF documents the design of snapshot. Jing Zhao and Tsz-Wo Sze from Hortonworks gave a great talk on the design of HDFS snapshots. The slides can be downloaded at here. The development of snapshot is tracked by HDFS-2802.
What’s the difference between Reliability, Durability, and Availability for data storage system?
Posted onSome important concepts in distributed system like Hadoop distributed file system, Google file system and so on. Answer from http://www.quora.com/Whats-the-difference-between-Reliability-Durability-and-Availability-for-data-storage-system The difference between durability and availability is fairly simple. Durability is about what happens when all power goes out everywhere. Has all data been written to stable storage that doesn’t require power (e.g. disk/flash), in
Read more
How to backup iPhone on Linux?
Posted onHow to backup iPhone on Linux? Not using iTunes or Windows. You can backup iPhone using libimobiledevice. By the time I replied here, it supports up to iOS 7. Please check this post for a tutorial. Actually, the methods introduced can be used on Linux systems with the libimobiledevice installed. Backup iPhone: idevicebackup2 backup ~/iPhoneBackups/
Read more
How to adjust the system partition (C:) size of Windows?
Posted onThe disk management tools of Windows can adjust it to some level. But there are more space available as far as I can tell. How to further adjust the system partition (C:) size of Windows? You may check these tools: EASEUS Partition Master (free) Includes Partition Manager, Disk & Partition Copy Wizard and Partition Recovery
Read more
How to upgrade Fedora 19 to Fedora 20 through the network?
Posted onHow to upgrade Fedora 19 to Fedora 20 through the network? The tool for Fedora to upgrade the distribution is FedUp. Check this page for how to upgrade Fedora 19 to Fedora 20: http://fedoraproject.org/wiki/FedUp#How_Can_I_Upgrade_My_System_with_FedUp.3F Be sure to check the bugs in Fedora 20 before the upgrading: http://fedoraproject.org/wiki/Common_F20_bugs#Upgrade_issues Generally, it contains 3 steps: Preparation # yum
Read more
Quick method to sniff the data left on a Linux VM in the public cloud
Posted onVM disks in public cloud may be shared by customers. This is privacy problem. Is there a quick method to sniff the data left on a Linux VM? One possible method: # dd if=/dev/xvda bs=1M | strings -n 100 > strings.txt Using the dd and strings which are pervasive on Linux systems.
Cache at Facebook
Posted onAbout caching system at Facebook. According to: https://www.facebook.com/notes/facebook-engineering/monitoring-cache-with-claspin/10151076705703920 Facebook has two major cache systems: Memcache, which is a simple lookaside cache with most of its smarts in the client, and TAO, a caching graph database that does its own queries to MySQL. The NSDI’13 paper introduces more about Memcache: https://www.usenix.org/conference/nsdi13/scaling-memcache-facebook The USENIX ATC’13 paper introduces
Read more
Systems Conferences
Posted onWhich ones are good systems conferences? Top ones by ACM and USENIX: OSDI: https://www.usenix.org/conferences/byname/179 SOSP: http://sosp.org/ Other SIGOPS Events: http://www.sigops.org/conf-sponsored.html EuroSys: http://www.eurosys.org/ SoCC: http://www.socc2013.org/ (SoCC 2013) ASPLOS: http://www.sigplan.org/Conferences/ASPLOS/Main VEE: http://www.sigplan.org/vee.htm USENIX ATC: https://www.usenix.org/conferences/byname/131 NSDI: https://www.usenix.org/conferences/byname/178 IEEE Conferences: ICDCS: http://www.temple.edu/cis/icdcs2013/ (2013) IPDPS: http://www.ipdps.org/ Other related ones and workshops: HPCA: Search HPCA ConferenceSC: http://www.supercomp.org/IEEE CLUSTER: http://www.clustercomp.org/ HotCloud:
Read more