How to find empty files in UNIX?

There are multiple ways to find whether a file is empty or not. Please see few examples below,

volcano@volcano-laptop:~/shellscript$ ls -ltr | grep '\<0\>'
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:41 sujay
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:42 jsk
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:45 a
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:46 b
-rw-r--r-- 1 volcano volcano    0 2011-04-10 22:30 emptyfile

volcano@volcano-laptop:~/shellscript$ ls -ltr | awk ' $5==0 {print}'
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:41 sujay
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:42 jsk
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:45 a
-rw-r--r-- 1 volcano volcano    0 2011-03-12 15:46 b
-rw-r--r-- 1 volcano volcano    0 2011-04-10 22:30 emptyfile

volcano@volcano-laptop:~/shellscript$ for fname in `ls`;do if [ ! -s $fname ];then ls -l $fname;fi;done
-rw-r--r-- 1 volcano volcano 0 2011-03-12 15:45 a
-rw-r--r-- 1 volcano volcano 0 2011-03-12 15:46 b
-rw-r--r-- 1 volcano volcano 0 2011-04-10 22:30 emptyfile
-rw-r--r-- 1 volcano volcano 0 2011-03-12 15:42 jsk
-rw-r--r-- 1 volcano volcano 0 2011-03-12 15:41 sujay

volcano@volcano-laptop:~/shellscript$ find . -maxdepth 1 -size 0 -ls
 58438    0 -rw-r--r--   1 volcano  volcano         0 Mar 12 15:42 ./jsk
 58434    0 -rw-r--r--   1 volcano  volcano         0 Mar 12 15:46 ./b
 58437    0 -rw-r--r--   1 volcano  volcano         0 Mar 12 15:41 ./sujay
 58211    0 -rw-r--r--   1 volcano  volcano         0 Mar 12 15:45 ./a
 58234    0 -rw-r--r--   1 volcano  volcano         0 Apr 10 22:30 ./emptyfile

How to remove empty lines from a file in UNIX or Linx

volcano@volcano-laptop:~/shellscript/empty$ cat testempty
hi first line

above line is empty
hi dude

above line is empty
volcano@volcano-laptop:~/shellscript/empty$ sed '/^$/d' testempty > testwithoutempty
volcano@volcano-laptop:~/shellscript/empty$ cat testwithoutempty
hi first line
above line is empty
hi dude
above line is empty
volcano@volcano-laptop:~/shellscript/empty$

volcano@volcano-laptop:~/shellscript/empty$ grep -v '^$' testempty > without
volcano@volcano-laptop:~/shellscript/empty$ cat without
hi first line
above line is empty
hi dude
above line is empty

One exception case is available for empty line removal. That is line will be empty but it will have space or tab characters. To handle that, we can use the below command format.


grep -v '^[]*]$'  oldfile > newfile

volcano@volcano-laptop:~/shellscript/empty$ cat test
hi first line

above line is empty
hi dude

above line is empty
   
volcano@volcano-laptop:~/shellscript/empty$ grep -v '^[         ]*$' test > newtest
volcano@volcano-laptop:~/shellscript/empty$ cat newtest
hi first line
above line is empty
hi dude
above line is empty
volcano@volcano-laptop:~/shellscript/empty$

volcano@volcano-laptop:~/shellscript/empty$ sed '/^$/d' test
hi first line
above line is empty
hi dude
above line is empty
   
volcano@volcano-laptop:~/shellscript/empty$ sed '/^[    ]*$/d' test
hi first line
above line is empty
hi dude
above line is empty
volcano@volcano-laptop:~/shellscript/empty$

How to remove CONTROL-M characters in UNIX or Linux

Control-m characters will get appended to a file when a file is transferred from windows to UNIX machine. There are multiple ways it can be removed.


Using vi editor:
:%s/^M//g

Using col command:
cat filename | col -b > newfilename

Using sed command:
sed 's/^M//g' filename > newfilename

Using dos2unix comand:
dos2unix filename newfilename

Examples:

volcano@volcano-laptop:~/shellscript/ctrl$ cat -v jsk
hi^M
jsk^M
volcano@volcano-laptop:~/shellscript/ctrl$ cat jsk | col -b > jsk.new
volcano@volcano-laptop:~/shellscript/ctrl$ cat -v jsk.new
hi
jsk
volcano@volcano-laptop:~/shellscript/ctrl$ sed 's/^M//g' jsk > jsk.new2
volcano@volcano-laptop:~/shellscript/ctrl$ cat -v jsk.new2
hi
jsk

Note: Hold the control key and then press v and m to get the control-m character

How to remove files starting with - minus in the filename

There might be several ways to remove files starting with first character as - minus. But i am providing 3 solutions below.

1. $ rm ./-filename

2. $ rm -- -filename

3. Find the inode number by using ls -i and then use the below command
find . -inum 45678 -exec rm {} \;

Difference between Unix Internal & External Commands

Internal Unix commands are built with the shell and hence it will run in the same shell.
eg: cd

External Unix commands will start a new shell and then it gets executed
eg: ls

Note: If you execute a shell script, it will create or fork a new shell and then execute the commands present in the script file. You can also execute the shell script in the same shell by using source or . infront of the shell script.

eg:
sujay@laptop:~/scripts$ cat test.sh
cd /
echo "Current dir is `pwd`"

sujay@laptop:~/scripts$ test.sh
Current dir is /
sujay@laptop:~/scripts$ pwd
/home/sujay/scripts
sujay@laptop:~/scripts$ source test.sh
Current dir is /
sujay@laptop:/$ pwd
/

How to run a unix shell script from any directory

To run a script from anywhere inside UNIX directory, you need to include the script absolute directory in the PATH variable. After that without using the pathname we can execute the script successfully.

eg:
In .profile, update PATH=$PATH:/home/sujay/scripts and source the .profile

Script Loc: /home/sujay/scripts/
Script Name; anywhere

sujay@laptop:~$ anywhere
++ pwd
+ echo 'Executed from /home/sujay'
Executed from /home/sujay

Order of execution for /etc/profile .bash_profile .bash_login .profile /etc/bash.bashrc .bashrc

If interactive login-shell or non-interactive login shell with --Login option is started, it will start reading & executing in the order /etc/profile, .bash_profile, .bash_login and finally .profile

If interactive shell, that is a non-login shell is started, it will start reading & executing in the order /etc/bash.bashrc, .bashrc

Difference between .profile and .bashrc

.profile will be read, whenever login-shell is started

.bashrc will be read, whenever new interactive shell(not a login shell) is started

How to compress multiple files using tar and gzip

1. Put all the required filenames in a file

2. tar all the files

3. gzip the tar file

eg:

vjsujay@gmail.com$ ls *.html > filelist

vjsujay@gmail.com$ fname=filelist

vjsujay@gmail.com$ tar -cvf multihtml.tar -L $fname

vjsujay@gmail.com$ gzip multihtml.tar

Final file will be multihtml.tar.gz

To display only the user directories in Unix or Linux

vjsujay@home$ ls -lA "pipeline symbol" grep '^d'
drwxr-s--- 2 n0160429 ccipadm 512 Jul 21 00:17 .temp
drwxr-s--- 2 n0160429 ccipadm 512 Mar 15 04:58 cal
drwxr-s--- 2 n0160429 ccipadm 512 May 06 04:50 dump
drwxr-s--- 2 n0160429 ccipadm 512 May 05 23:14 script
drwxr-s--- 2 n0160429 ccipadm 512 Jul 18 12:28 temp

Note: If you use ls -la means, you will get . (dot) and .. (dot dot) directory also.

Difference between single square and double square brackets in conditional check in shell scripting

1. [ conditional check ] -> Variables used in the conditional check should have value, otherwise exception will be thrown during the execution.

2. [[ cond check ]] -> Exception will not thrown, even if variable don't have value or not received any value during the execution.

So, it is advisable to use single square bracket always to ensure whether we are receiving the value and doing the conditional checking.

example:
echo "Enter your name: "
read NAME
if [[ $NAME == "sujay" ]]
then
echo Your name is $NAME
else
echo Your name is $NAME from else part
fi

$ ksh doublesquare.sh
Enter your name:
sujay
Your name is sujay

$ ksh doublesquare.sh
Enter your name:

Your name is from else part

Note: If double square bracket is replaced with single: if [ $NAME == "sujay" ]
$ ksh singlesquare.sh
Enter your name:
sujay
Your name is sujay
$ ksh singlesquare.sh
Enter your name:

singlesquare.sh[4]: test: 0403-004 Specify a parameter with this command.
Your name is from else part

Different type of Command Substitution in Shell Scripting

In two ways we can do the command substitution.
1.) Using backquote: ``
2. Using dollar sign and parenthesis: $()
eg:
vjsujay@home $ echo My system name is `uname -n`
vjsujay@home $ echo My system name is $(uname -n)

To find last day of any month & year using unix command

How to keep track of your Unix/Linux command execution and its output

By using "script" command, you can keep track of Executed Commands and its output.

Definition: The script command will keep a transcript(written record) of everything you say to the computer and everything that the computer responds to you.

Syntax: script [capturefilename]

*) To start the script command
eg: vjsujay$ script cmdlog

Note: cmdlog will be created in the current dir and also if you are not specifying the filename means, it will create a default filename 'typescript'

*) To stop capturing or script command,
eg: vjsujay$ exit

Different ways of running a Shell Script

1. Executing the Shell Script file as a command in the Shell Prompt
vjsujay@linuxdemon$sample.sh

2. Executing the Shell Script file using specified Shell
vjsujay@linuxdemon$bash sample.sh

3. Executing the Shell Script file in the current Shell
vjsujay@linuxdemon$source sample.sh
or
vjsujay@linuxdemon$. sample.sh

4. Executing the Shell Script file using exec command
vjsujay@linuxdemon$ exec sample.sh

Copying longfilenames without retyping

$ ls -l thisismytestfile
-rw-r--r-- 1 Ajay None 13 Apr 17 03:22 thisismytestfile

Ajay@intel ~
$ cp thisismytestfile{,.bk}

Ajay@intel ~
$ ls -l thisismytest*
-rw-r--r-- 1 Ajay None 13 Apr 17 03:22 thisismytestfile
-rw-r--r-- 1 Ajay None 13 Apr 17 03:26 thisismytestfile.bk

How to Truncate a file in Unix/Linux

:>filename

If the filename already exists,the file will be truncated to ZERO byte file else a new file with ZERO byte will be created.

Example:

[vjsujay@phenix vjsujay]$ ls -l jsk
-rw-r----- 1 vjsujay free 46 Jun 25 20:14 jsk

[vjsujay@phenix vjsujay]$ : > jsk

[vjsujay@phenix vjsujay]$ ls -l jsk
-rw-r----- 1 vjsujay free 0 Jun 25 20:15 jsk

One more example with 3 different methods:

volcano@volcano-laptop:~/test/trunc$ ls -ltr
total 12
-rw-r--r-- 1 volcano volcano 14 2011-02-23 11:56 rado
-rw-r--r-- 1 volcano volcano 14 2011-02-23 11:57 radoo
-rw-r--r-- 1 volcano volcano 14 2011-02-23 11:57 radooo
volcano@volcano-laptop:~/test/trunc$ > rado
volcano@volcano-laptop:~/test/trunc$ :>radoo
volcano@volcano-laptop:~/test/trunc$ true > radooo
volcano@volcano-laptop:~/test/trunc$ ls -ltr
total 0
-rw-r--r-- 1 volcano volcano 0 2011-02-23 11:57 rado
-rw-r--r-- 1 volcano volcano 0 2011-02-23 11:57 radoo
-rw-r--r-- 1 volcano volcano 0 2011-02-23 11:57 radooo

Commenting Multiple lines in a shell script

By using anonymous here document, we can comment multiple lines in a shell script for debugging purposes.

eg:

: << COMMENT
echo "Not in the Scope of debugging/execution"
a=5
b=3
let c=a+b
echo $c
COMMENT

The statements in between the anonymous here document will not be executed.

Length of a string in Linux

The command syntax is ${#string}

Example:
linuxdemon$lindem="linux demon"
linuxdemon$echo ${#lindem}
11

Explanation:
1.Assigning the string linux demon to the variable lindem.
2.Determining and printing the length of the string.

FTP Automation using Shell script

I am going to give the code in shell script for ftp automation.

ftp -n servername/serverIP << FTP_AUTO
user giveusername givepassword

Any ftp command you can give here

bye
FTP_AUTO

Explanation:
1. -n is used to restrict the ftp from attempting auto login.
2. << here document concept applied here and hence the tag is FTP_AUTO
3. user is the ftp command
4. You can give any ftp command after the user command to do your job.
5.End the here document with the same tag FTP_AUTO

How to mount WIN partition in Linux?

Temporary Mounting: After the reboot of your system, you need to again execute the mount command to mount the win partition in Linux.

1.You need to create mount point(directory) for each win partition(drives) inside the /mnt directory.
eg: mkdir /mnt/winc (You can give any name instead of winc)

2. By using the mount command, we can now mount the windows partition
eg: mount -t vfat /dev/hda1 /mnt/winc (In linux, first partition is called as hda1)

3. Now get inside the /mnt/winc directory and check the files.You will be able to see your files from C drive.
eg: cd /mnt/winc

Permanent Mounting: To do the permanent mounting, you need to follow these steps below.

1.Create mount points (mkdir /mnt/winc)

2.Edit the /etc/fstab file using any text editor like vi and insert the mount point and file system details at the end of the file with the proper spacing as previous lines.
eg: /dev/hda1 /mnt/winc vfat

3. After the reboot of your system, you will be able to access all C drive files.

How to rename group of files?

I am giving a simple shell script, for renaming a group of files in the current directory.In the script, i have renamed all .jsk files to .txt files.

for filename in *.jsk
do
mv $filename `basename $filename jsk`txt
done

Explanation:
1.In the for statement, i am finding out all the .jsk extension files.
2.In the mv statement, i am using basename command to strip the jsk extension.