Bash Rules

Whats your background?

2007-07-03T00:20:00.001-07:00

As part of job control its possible to stop a process and continue it later.
So while you are running a program, you can type Ctrl-Z to suspend it , its like a pause in a media player

Then you can ask bash to continue the process in background by typing 'bg'.
Background processes can not read from terminal and if they try to they will be suspended(paused).

Here is quick test for this:


bash$ cat read.sh
#!/bin/bash

sleep 5
echo -n "Enter some text > "
read text
echo "You entered: $text"

Run the script and hit Ctrl-Z


bash$ ./read.sh

[1]+  Stopped                 ./read.sh

Now run this in background using bg command.

You are prompted to enter some text


bash$ bg
[1]+ ./read.sh &
bash$ Enter some text >

As soon as you enter some text and hit Enter you will see this


bash$ Enter some text >

[1]+  Stopped                 ./read.sh

Any more attempts to run this in backgroup will suspend the script.
Run fg to get the script to run in foreground to get out of this loop.


bash$ bg
[1]+ ./read.sh &

[1]+  Stopped                 ./read.sh
bash$ bg
[1]+ ./read.sh &

[1]+  Stopped                 ./read.sh
bash$ bg
[1]+ ./read.sh &

[1]+  Stopped                 ./read.sh
bash$ fg
./read.sh
hi there
You entered: hi there

Reviewing Sed - Part 1

2006-12-27T22:49:00.000-08:00

There is a good collection of sed one liners at http://www.student.northpark.edu/pemente/sed/sed1line.txt.

So I have just begun to review few them using sed manpage because of the earthquake in Taiwan.

# Following line will insert a blank space after everyline in temp.txt


 sed G temp.txt

G - Append hold space to pattern space.

So hold space holds nothing initially and if you append that to pattern space, you get a new line.

So thats the way to add blank lines after everyline.
Sed goes through each line and loads it to the pattern space.

# Adds only a single blank line to all lines to temp.txt


sed '/^$/d;G' temp.txt

/regexp/ - Match lines matching the regular expression regexp.

d - Delete pattern space. Start next cycle.

There are two things to consider.
For blank lines : 'd' is executed that deletes the pattern space and starts next cycle(loads next line and starts all over again)

For non blank lines : as the pattern doesn't match, d is not executed and next command G gets executed.
We get a blank line after every non blank line.

The command would delete the blank lines and skip the G command for blank lines.
Thing to note is that d command applies only to lines matching the regex /^$/ whereas
G command applies to all lines and is not tied to the regex. However when regex is matched the d command will cause G command to be skipped.

Sed and No DuPlication

2006-12-13T00:23:00.000-08:00

I needed to remove duplicate names from a file so I thought sed should be a good choice for it.
uniq is too easy so I found out how sed does it. Herez how


$  sed '$!N; /^\(.*\)\n\1$/!P; D' filename

So now time for an explanation.

1 $!N - sed reads one line at a time and then works on it. It doesn't read in the newline at the end of the line into pattern space (what sed works on). So N command appends the newline and the next line to the pattern space.
$ denotes the last line and ! means NOT, so it means for the last line don't execute N command.Heck as if there is anything to read after the last line..read it if you can for all I care.

2. /^$.*$\n\1$/!P; - If you see start of pattern space and then anything followed by a newline which is followed by exactly that anything. Don't "print the first part of pattern space till the newline"(P) else print first part of pattern space till the newline

3. D - "just delete the first part of pattern space till the newline and restart the command cycle i.e go back to N"

Its time for tee

2006-12-08T00:19:00.000-08:00

I was asked this question and suddenly i realized that bash doesnt have redirection like operator to do this. Do what? Print Standard Output/Error as it is and also redirect it to a file. So therefore its time for tee.


$ls i_dont_exist 2>&1 | tee output.txt
ls: i_dont_exist: No such file or directory
$cat output.txt
ls: i_dont_exist: No such file or directory

Colorful Bash

2006-10-19T03:08:00.000-07:00

Well here is how(er..i mean how not to :)) to make your terminal a bit more colorful


[kalyan@duck snippets]$PROMPT_COMMAND='tput setf $(( RANDOM % 7 + 1 ))'

And so we get a colorful terminal

Bash Meditation

2006-07-14T00:52:00.000-07:00

Here is the oneliner for meditation in bash

while :; do whoami; done

First post using Flock

2006-05-15T21:10:00.000-07:00

Let me see :)

CSS Woes!!!

2006-04-19T02:45:00.000-07:00

Well spent a good deal of time with Prateek debugging an issue with links not getting underlined
when <a name= is used and ended up with this

Migrate apps from Internet Explorer to Mozilla

Thank you guys :)

Bash : Order of expansions

2006-04-16T01:27:00.000-07:00

This one is an important thing to understand for anyone who wants to know how bash works.
So here is a refresher, this is the order of expansions:
* brace expansion
* tilde expansion
* parameter and variable expansion
* command substitution
* arithmetic expansion
* word splitting
* filename expansion
Now here is what i am going to do,go through all of these and actually see if they work as they should.
First lets check the first two : brace expansion and tilde expansion

kalyan@thunderbird:~$ echo ~ #tilde expansion:current users homedir
/home/kalyan
kalyan@thunderbird:~$ echo ~b  
~b
#tilde expansion fails: no user with name b
kalyan@thunderbird:~$ echo ~kalyan 
/home/kalyan
#tilde expansion works and gives kalyan user's home dir
kalyan@thunderbird:~$ echo a{b,,c} #brace expansion
ab a ac
kalyan@thunderbird:~$ echo ~{b,,kalyan} #test to see who comes first
~b /home/kalyan /home/kalyan
kalyan@thunderbird:~$ echo /home/kalyan{b,,kalyan} 
/home/kalyanb /home/kalyan /home/kalyankalyan
#brace expansion first as otherwise we'd get

Now time for parameter and variable expansion. This is simple to check we add a variable


kalyan@thunderbird:~$ myvar=kalyan
kalyan@thunderbird:~$ echo $myvar
kalyan
kalyan@thunderbird:~$ echo ~{b,,$myvar} 
~b /home/kalyan ~$myvar
#strange, no variable expansion
kalyan@thunderbird:~$ echo ~{b,,"$myvar"} 
~b /home/kalyan ~kalyan
#variable expansion but only after tilde expansion

So we need to enclose the variable in quotes to force bash to expand in this scenario.
Okay lets move on to command substitution now


kalyan@thunderbird:~$ cat > myfile 
kalyan
#create a file with contents as my name
kalyan@thunderbird:~$ cat myfile #
kalyan
kalyan@thunderbird:~$ myvar1=myfile
#create a variable with value myfile 
kalyan@thunderbird:~$ cat $myvar1
kalyan
kalyan@thunderbird:~$ echo $(cat $myvar1) 
kalyan
#shows myvar1 is expanded before command substitution
kalyan@thunderbird:~$ echo ~{b,,$(cat $myvar1)} 
~b /home/kalyan ~$(cat $myvar1)
#so no var expansion or command substitution
kalyan@thunderbird:~$ echo ~{b,,"$(cat $myvar1)"} 
~b /home/kalyan ~kalyan
#we put quotes and now it works

So that proved that command substitution follows variable/parameter expansion. Now its arithmetic expansion


kalyan@thunderbird:~$ echo $(( 5 + 5 )) #arithmetic expansion
10
kalyan@thunderbird:~$ cat > five
5
kalyan@thunderbird:~$ cat five
5
kalyan@thunderbird:~$ echo $(( $(cat five) + $(cat five) )) 
10
#we should get 10. :)

That was neat. wasnt it :)..now time to get little insane. Bash splits words using the characters in IFS as the seperator so lets engineer an example to prove that word splitting happens after arithmetic expansion ;)


kalyan@thunderbird:~$ echo $(( 101010101 + 202020202  ))
303030303
kalyan@thunderbird:~$ OFS=$IFS #save the IFS
kalyan@thunderbird:~$ IFS=3 #change it
kalyan@thunderbird:~$ echo $(( 101010101 + 202020202  ))
 0 0 0 0
#wow where did the 3's go, they were removed during word splitting :)
kalyan@thunderbird:~$ IFS=$OFS #back to sanity

Now we are onto the last expansion and thats filename expansion. bash actually looks for files to expand '*','?'.


kalyan@thunderbird:~/fun$ cat > 10_file_name_starts_with_ten
hmm i am tired of bash expansions now :)
kalyan@thunderbird:~/fun$ echo 10*
10_file_name_starts_with_ten
kalyan@thunderbird:~/fun$ IFS=2 #for the example below
kalyan@thunderbird:~/fun$ echo $(( 30*7 ))*
 10_file_name_starts_with_ten
#30*7 expands to 210 then bash removes '2' because of word splitting,that leaves us with 10* which undergoes filename expansion ..so simple :P
kalyan@thunderbird:~/fun$ IFS=$OFS #restore sanity

Thats it and i really mean it when i say

 hmm i am tired of bash expansions now :)

Why gmail sucks

2006-04-05T01:37:00.000-07:00

Have a look at its Received Header in More Options , Show Original


Received: by 10.70.90.15 with SMTP id n15csd722wxb;
        Thu, 9 Mar 2006 00:46:04 -0800 (PST)

Now why would I care for a stupid mail server's internal i.p?

Firefox cookie policy

2006-04-04T23:25:00.000-07:00

Firefox (1.5,1.0.7) lets you set a domain level cookies for top level domain for example ac.uk.
IE blocks these cookies but firefox allows them,so which one is better ? :)

Key findings

2006-03-24T00:11:00.000-08:00

Haven't blogged bash for a while but then BASH is good :)

Problem: Application A has many config files ( only mutiple lines of key=value type), so one big config file is created merging all the ones(in a very randomized fashion). Now how do you verify that the merged configuration file has not missed out anything.

Solution: A single line of bash code :)


for i in small* ; do while read new; do grep -q ${new%=*} big.cfg || echo ${new%=*} of file $i not found in big.cfg ; done < $i ;done

Formula One

2006-03-20T04:29:00.000-08:00

Spent this weekend in KL (Kuala lumpur) with some friends from Thoughtworks ..Gabbar..Dator..S2m8kh..Little Chaman..Superman. Had great fun and in the process learnt a thing or two about F1..which I intend to unlearn soon :)...Dator looks crestfallen after Kimi's exit :)

List assignment in scalar/boolean context in perl

2005-09-22T03:01:00.000-07:00

Though this one is about perl but as i havent blogged for sometime
I'll post it here.

A collegue of mine was using a contruct like

while(my $temp = $sth->fetchrow_array)
{
# do work..

}

The problem was that whenever fetchrow_array returned 0 as
the value ,the while would exit out.

So the solution suggested by another colleague was to enclose $temp in a list
such as (my $temp)=$sth->fetchrow_array.

I was intrigued by this as a list should return the last element in scalar
context.

As it turns out
while((0)) wouldnt enter the while block but strangely.
while(($temp)=(0)) works.

This is because a list assignment in scalar context returns the number of
elements on the r.h.s of the assignment.

Yup there is more than one way to do it and one could also use
while(defined($temp=(0))) ..

Removing newlines from a file using sed

2005-08-17T01:01:00.000-07:00

I kept forgetting this as I never really realized that sed is a line editor.
So the newline at the end of the line isnt available by default unless we create a multiline pattern space using N.

Anyways here is the sed command that does the magic.

$sed ':a;N;$!ba;s/\n//g' filename

Among other things it uses labels and N command. :)

Python regex to match a floating point number

2005-08-12T02:24:00.000-07:00

Here is the expression to match a float upto two places of decimal.
((?<\.|\d)\d+ (?:\.\d{1,2})?)So let me explain it in steps:

'(?<!..)' is a negative look behind,i.e,at that position the preceding text should not match the regex enclosed in '(?<! )'.In this case it is '\.|\d' which stands for a literal dot or a digit.This is not part of the match,the regex that needs to match starts with '\d+'.
So what we are saying is match 1 or more digits but make sure those digits are not preceded by a literal dot or a digit. Well one can understand why digits shouldnot be preceded by a literal '.' (dot) ,for example we dont want .99 to match but why do we need the digit part. The reason is quite subtle but without it .99 will be matched by the regex.

This is because when regex engine will try to match .99 initially it will fail \d+ matches 99 but as it is preceded by negative lookbehind for a literal '.',the match cant succeed so the engines shifts to next character and makes \d+ match only the rightmost 9 in 99.
Also as the rightmost 9 is preceded by a 9 the negative look behind is also satisfied so \d+ will be end up matching just the 9.
This is a false positive as we definitely dont want to match that so we make the negative look behind include a digit to rule out this result

Bash arrays

2005-08-12T02:22:00.000-07:00

First off inside bash arrays ,elements are separated by space.
Have a look at this.

$cat new.sh
length=1
positions=(0 1 2)
positions[length+1]=4
echo "${positions[*]}"

./new.sh
0 1 4

Point to note is that 'length' as well as '$length' areboth valid indexes.

bre,ere and pcre

2005-08-11T02:41:00.000-07:00

Its been a while since I had a look at "Mastering Regular Expressions" by Jeff Friedl but I am glad i went through it once.

So the input is like

filename="http://really.big.url/test?foo=bar"

So my first attempt at extracting the url was

$cat output| sed 's/^.*filename=//'

So someone pointed out that the url could contain 'filename=',yeah thats quite possible and proposed a solution with back-references as sed doesnt have non greedy quantifiers which are common to pcre(perl compatible regular expressions).Sed only supports bre(basic regular expressions) and ere(extended regular expression).Dont ask me what they are :).



$cat output| sed -r 's/^[[:space:]]*filename="([^"]*)"[[:space:]]*$/\1/g

In addition to making space for "spaces" :),this moves the greedy "*" to the left hand side of "filename=" so that any extra "filename=" are matched by the greedy "*".

The same thing in perl could be simply done by

$cat output|perl -wpe 's/^.*?filename=//;'

Limits of sed

2005-08-10T07:18:00.000-07:00

Well a user had a huge ascii file 1.5 gigs and all of that on a single line.Wow!

So the job was to make it a span many lines,by putting a newline after every 310 characters.

So the suggested sed expression was:

$sed 's/$.\{310\}$/\1\n/' filename

But it appears that sed crapped out with following error:

sed: Couldn't re-allocate memory

Maybe something like C is better suited for this as sed is basically a line editor,so maybe tries to read and operate on it in a single go :).

A bit of sed and awk magic

2005-08-10T02:27:00.000-07:00

The task is to find the total cpu utilization of httpd processes.
Instead of doing all kinds of acrobatics with grep,bc and bash loops , one can simply use awk.

$ps aux | awk '/httpd/{s+=$3}END{print s}'

One a side note , gnu sed can change a file in-place .This is done by using the "i" option.

$sed -i 's/fun/bun/' test.txt

Disable enter key in bash

2005-08-04T02:11:00.000-07:00

Well this is very straightforward, we just need to unbind keybinding for
"\r".Thats easily done using bind.

bind -r '\r'

Now to get the binding back i used.

bind '"\r":accept-line'

Obviously i had to hit Ctrl-J instead of "Enter" :).

Ctrl-XCtrl-V

2005-07-28T06:29:00.000-07:00

Well i just bumped into this article (http://www.chiark.greenend.org.uk/~sgtatham/bash-notify/)
searching for aynchronous jobs.
As my brain is a non starter i dont understand some terms like asynchronous.
So now to keep things simple i feel an asynchronous job is a background job as it can end anytime :).

According to the article "set -b" and "set +b" both of them have issues.
set -b will just corrupt your display and "set +b" can create lot of jobs.Dont ask me why :).
But the feature of Ctrl-XCtrl-V prints the bash version etc and then restores the command line on new line "as it was".

$ps ax
GNU bash, version 3.00.16(1)-release (i686-pc-linux-gnu)
$ps ax

The author had even written a patch for this ..Will have a look at it sometime :).

hash builtin - 2

2005-07-28T03:29:00.000-07:00

Okay i am back to the hash builtin again.Lets trace phash_flush in hashcmd.c.
As expected it calls hash_flush.A look at the top of the function shows what it does,removes and discards
all entries in TABLE.If FREE_DATA is non-null then its kind of a destructor else call free() to free up the memory.

First it is checked if (table == 0) or if number of entries in the table are zero.

Once we are sure that number of hash entries are not zero, we iterate through the hash tables buckets.
Typically we call free(item->data),free(item->key) and free(item).

And finally we set the initial bucket pointer to NULL and set the number of entries to zero.

This is quite staightforward ,the way it should be.
So as we are doing okay ,lets move to the other functions.
Lets see phash_remove now,this as expected calls hash_remove.
A peek inside the code shows we try to get the bucket for the string first.
This buckets is got by hashing the string and anding with number of buckets -1 :).
The string typically is 'ls','ps' etc no mention of the data here but then who cares when its is getting deleted.
No.The pointer is being returned after being deleted from the list so lets go back to phash_remove.
Well i went back to phash_remove and tried to print the data before it gets freed.No luck.
I am reading junk so where is the path stored??
Okay i screwed up again ,without reading i assumed the complete file path would be stored as item->data but
thats not the case.item->data is a pointer to a struct PATH_DATA which contains the path.

Didnt i tell you my brain is a goner anyway back to code and let me print the pathname just before it gets freed,
(Dead Man Walking style) :).

Bingo.. now i get to print /usr/bin/ls just before its cleaned up by "hash -d ls" :).

hash builtin

2005-07-27T06:28:00.000-07:00

I think my brain will finally be a non starter.Anyhow here are my thoughts on the hash builtin of bash.

So i browsed haslib.h where HASH_TABLE is defined as

typedef struct hash_table {
BUCKET_CONTENTS **bucket_array; /* Where the data is kept. */
int nbuckets; /* How many buckets does this table have. */
int nentries; /* How many entries does this table have. */
}

Obviously the next question would be what is BUCKET_CONTENTS,remember a hash has buckets..
Well not quite..:) anyways as we have the source with us lets go there ..:)

typedef struct bucket_contents {
struct bucket_contents *next; /* Link to next hashed key in this bucket. */
char *key; /* What we look up. */
PTR_T data; /* What we really want. */
unsigned int khash; /* What key hashes to */
int times_found; /* Number of times this item has been found. */
} BUCKET_CONTENTS;

As we can see hash_table has bucket_array,number of buckets and number of entries.
I have feeling the number of buckets may increase as we need more...
As we look into BUCKET_CONTENTS has a pointer to the next bucket,
a key ( like 'ls'),data (/usr/bin/ls),khash ( i think this is the hashed value),
times_found ( lol..source is enough i guess :) ).

Now lets look at the functions in hashcmd.c

phash_create
phash_freedata
phash_flush
phash_insert
phash_insert
phash_search

phash_create
This in turn calls hash_create (hashlib.c) with appropriate number of buckets (FILENAME_HASH_BUCKETS=64)

phash_search
Probably this is what is interesting,this in turn ccalls hash_search.

bucket= HASH_BUCKET(string,table,hv)

this in turns calls hash_string,the real hashing function.

for (i = 0; *s; s++)
{
i *= 16777619;
i ^= *s;
}

The comment says magic is in the interesting relationship between the special prime 16777619 and 2^32 and 2^8.
As fas as i can tell,2^8 is *s and i is 2^32 :).

HASH_BUCKET ands the value of string with the number of buckets.
I am just thinking of collisions but i guess each bucketjust starts up a link list.

spawnve

2005-07-27T03:36:00.000-07:00

There was this patch on the bash mailing about making bash run faster on cygwin.

It was pointed out that there were issues with signal handling after the patch is applied.
After the function in the patch is called SIGINT handler is getting lost making the shell to exit instead of just cancelling the current line.

A quick look at the patch shows the following code.

+ sigdelset(&child_sig_mask, SIGTSTP);
+ sigdelset(&child_sig_mask, SIGTTIN);
+ sigdelset(&child_sig_mask, SIGTTOU);

Job control signals are removed from the child_sig_mask,so does that mean they will not be blocked?
In any case the mask was set to top_level_mask instead of child_sig_mask in the following line and was pointed out.

+ sigprocmask (SIG_SETMASK, &top_level_mask, &old_sig_mask);

Quoting during command substitution

2005-07-22T03:54:00.000-07:00

I was corrected in #bash channel in irc by trash regarding this.

During variable assignment,its better to quote the rhs.

For example

$myvar="hello world"

but this isnt needed if you are doing something like

$myvar="$(echo hello world)"

As this is equivalent to

$myvar=$(echo hello world)

A small thing about vi regex

2005-07-14T03:29:00.000-07:00

Well i was trying to remove all the spaces in the lines that are present before text.

So i thought to use vi's command mode and run


:%s/^\s+//

But this doesnt work ,then i found out (http://www.geocities.com/volontir/) that in vi the way to specify the '+' quantifier is to use,
'\+'.So it turns out the way to do it is .


:%s/^\s\+//

Also '?' quantifier is '\='.

PROMPT_COMMAND

2005-07-11T06:15:00.000-07:00

Its an interesting bash variable..bash whenever it finishes a command or null command would try to run this..


$ PROMPT_COMMAND=date
Mon Jul 11 18:46:42 IST 2005
$
Mon Jul 11 18:46:42 IST 2005
$
Mon Jul 11 18:46:43 IST 2005
$
Mon Jul 11 18:46:43 IST 2005
$ uname
Linux
Mon Jul 11 18:47:23 IST 2005

I guess it will be more fun to set it to "clear" :).

completion for directories starting with '='

2005-07-11T02:46:00.000-07:00

Initially i thought this could be a bug but later realised that this behavior is because of readline treating '=' as a word break character.


# mkdir =foo =foobar
# cd =\=foo
-bash: cd: ==foo: No such file or directory


Basically when you hit TAB after types just '=' , bash adds "\=foo".

What is treated as a word break character can be modified by changing the value of COMP_WORDBREAKS(Thanks to Chet :) ).


$ echo $COMP_WORDBREAKS
"'@><=;|&(: 
$ COMP_WORDBREAKS=${COMP_WORDBREAKS/=/}
$ echo $COMP_WORDBREAKS
"'@><;|&(:

Also you just saw bash's search and replace functionality.If there is no "slash" in the end ,the pattern is deleted from the variable :).

filename expansion after parameter expansion

2005-07-06T00:03:00.000-07:00

This point came up during an intersting post on bug-bash.I just ran a few commands to check this out.

If nullgob is set then bash will not echo the pattern if it fails the filename expansion.


$echo abc*
abc*
$shopt -s nullglob
$echo abc*

$

So now if you have a variable like

$test=p*
$echo $test
purchase
$echo "$test"
*

Moral of the story:Quoting (double) a variable suppresses filename expansion.

Three arrows

2005-07-03T22:21:00.000-07:00


$ cat <<< Yothere
Yothere

cat <<< "$(< for.sh)"
#for(( i=1;i<6;i++))
for i in $(ls)
do
echo $i
done

I can call it extended here document .So as you can see this allows for variable subsitution as well. :)

Mama i am coming home..

2005-06-23T23:24:00.000-07:00

Some times mama maynot be back home to receive you ,in that case you become a zombie .Mama need to call wait to make sure all the kids are back.

%cat checkzombie.sh
#!/bin/bash

./zombie1 &
./zombie
wait
%cat zombie1
#!/bin/bash

sleep 60
%cat zombie
#!/bin/bash

sleep 2

If you dont call wait at the end,the script will return before zombie1 is done as
zombie was executed as a background job.

Job Control

2005-06-23T02:17:00.000-07:00

Job control lets you suspend _lot of jobs and then bring them into foreground.
Ctrl-Z takes the job into background...
But you can get it back by 'fg %n' where n is the ID of the job.

$ jobs
[1]- Stopped ./trace.sh
[2]+ Stopped ./trace.sh
$ fg %2
Coming soon are the traps in bash..beware..:)

A text browser with links --dump

2005-06-22T23:53:00.000-07:00

This browser implemets quit (q) , back (b) and url.And a hidden command called 'd' which dumps history.More importantly i use the bash arrays.On L.H.S its HISTORY[$i] where as on R.H.S its ${HISTORY[$i]} and the length of array is ${#HISTORY[@]}...


#!/bin/bash
HISTORY=()
BACKINDEX=0

function updatehistory() {
    numvalues=${#HISTORY[@]}
    if (( $numvalues < 3 ))
        then
            HISTORY[$numvalues]=$1
        else
                    index=$(( numvalues -1))
            for (( i=0;i<$index;i++))
            do
                         j=$(( i+1))
                     HISTORY[$i]=${HISTORY[$j]}
                    done
                    i=$(( i++))
                HISTORY[$i]=$1
            fi
}



function dump_history(){

numvalues=${#HISTORY[@]}

        for (( i=0; i < $numvalues; i++))
        do
                    echo ${HISTORY[$i]}
        done
  }

while true
do
        echo "Enter a url.q for quit and b for back"
        read choice

        case $choice in

            q) exit ;;

            b) BACKINDEX=$(( BACKINDEX+1))
            numvalues=${#HISTORY[@]}
              INDEX=$(( numvalues - BACKINDEX -1 ))
              links  -http-proxy proxyserver:8088  -dump ${HISTORY[$INDEX]}
              ;;

            d) dump_history;;

            *) links  -http-proxy proxyserver:8088  -dump $choice
                        if ((  $? == 0 ))
                then
                    updatehistory $choice
                        fi
                        BACKINDEX=0
                        ;;
        esac

done

Counting lines with grep

2005-06-22T03:28:00.000-07:00

Interesting option called -c , which lets you suppress output and oonly print the count of lines that matched so here is a replacement for "wc -l".

%cat bajajautofinance |grep -c ^.*$
11
%cat bajajautofinance |wc -l
11

A menu based bash directory browser

2005-06-21T02:42:00.000-07:00

This could be useful when you have long pathnames to browse around and these pathnames have similar names...so CDPATH may not work.


#!/bin/bash

function cd()
{
if [ -z "$1" ]
then
    pushd $HOME
    return 0
fi
THEDIR=$(echo $1|sed -e "s/\~/$(echo $HOME|sed -e 's/\//\\\//g')/g")
pushd $THEDIR >/dev/null 2>&1


}

function lsd()
{
if [ -z "$1" ]
then
    select mydir in $(dirs -v|awk '{print $2}'|sort -u)
        do
        cd $mydir
        break
        done
else
        if [ -d "$1" ]
        then
            select mydir in $(find $1 -type d)
                do
                    cd $mydir
                    break
        done
    fi

fi

}

Blocks and IO

2005-06-17T00:07:00.000-07:00

{

echo "Hello World"

echo "Bye World"

} > output.txt

And here is some bad news..Daniel Robbins(Gentoo Founder) joins MS...isn't that cheap ;)

Basic usage of select

2005-06-13T21:32:00.000-07:00

Select creates a menu for you and its very simple to use.Here is an example

#cat select.sh
select test
do
echo $test
done
#./select.sh item1 item2 item3
1) item1
2) item2
3) item3
#? 2
item2
1) item1
2) item2
3) item3

Also this creates an infinite loop...so you need a break :)

Using eval

2005-06-10T06:41:00.000-07:00

Here is some recursive(?) cool usage of eval.Thanks to Chandra..

$p=i
$i=1
$CM1=World
$CM=Hello
$eval HOST=\$CM\$$p
$echo $HOST
Hello1
$eval HOST=$CM\$CM$i
$echo $HOST
HelloWorld
$eval HOST=\$CM$i
$echo $HOST
World

Bash string operators

2005-06-10T03:05:00.000-07:00

#!/bin/bash

while read foo
do
echo ${foo%%:*}
case ${foo%%:*} in
cm1*) export CM1=${foo##*:};;
esac

done < config.txt

echo $CM1

And the config.txt has syntax like
tunnel_port : 9999

Enjoy :)

Alternative for loop syntax

2005-06-07T03:08:00.000-07:00

for(( i=1;i<6;i++))
do
echo $i
done

bash version of colorise python script

2005-05-31T22:18:00.000-07:00

Well not exactly it only does a word not a string but i guess you get the idea.
Here is the python version.
script in python


#!/bin/sh

col=""
function colorize {
            case "$1" in   
                "red" )
                                   col=1;;
                    "green" )
                                   col=2;;
               "yellow" )
                                  col=3;;
                   "blue" )
                               col=4;;
                     * )
                          echo "wrong color dude"
                          exit;;
          esac

returnstring="\033[;3${col}m${2}\033[0m"
echo -e  $returnstring

}

strings=($*)

for(( i=1;i<$#;i++))
do
colorize $1 ${strings[$i]}
echo -en " "
done
echo

Searching forward through history

2005-05-31T06:32:00.000-07:00

Well the key to do is ctrl-s but thats already bound to stop for the tty.
So we undef that and use it :).

$cat .bash_profile |grep stty
stty stop undef start undef
(We press ctrl-s )
(i-search)`':

Substitution in the previous command

2005-05-31T06:29:00.000-07:00

Thanks to nujoom,there is one more bash trick.Only thing it works for the immediately previous command.The following sequence would end up opening the file in 'vi'.

$ls bajajautofinance
bajajautofinance
$^ls^vi
vi bajajautofinance

shopts

2005-05-29T23:29:00.000-07:00

#shopt -s cdable_vars
#fd=/home/kalyan/fun/bash/
#cd fd
/home/kalyan/fun/bash/
#pwd
/home/kalyan/fun/bash

Learning the Bash shell

2005-05-29T23:06:00.000-07:00

Got my hands on the third edition of this book...
Really good.
So for sometime my glob would include stuff i try out reading this book..

So here is something on aliases and options (set -o)

$set -o ignoreeof
'if you type ctrl-d now'
$Use "logout" to leave the shell.
$set -o noclobber
$echo hi > temp.txt
$cat temp.txt
hi
$echo sddsshi > temp.txt
-bash: temp.txt: cannot overwrite existing file
$echo sddsshi > temp.txt
-bash: temp.txt: cannot overwrite existing file
$echo sddsshi > temp.txt
-bash: temp.txt: cannot overwrite existing file
$rm temp.txt
$set -o noglob
$ls *
ls: *: No such file or directory
$set -o ignoreeof
$set -o nounset
$ls $ASD
-bash: ASD: unbound variable
$cd mybash
-bash: cd: mybash: No such file or directory
$alias
alias links='links -http-proxy rio:8088'
$alias mybash=/home/kalyan/fun/bash/
$cd mybash
-bash: cd: mybash: No such file or directory
$alias cd='cd '
$cd mybash
$pwd
/home/kalyan/fun/bash

Routing for localhost

2005-05-27T05:13:00.000-07:00

Cant believe it but needed this to be done to ping localhost on vmware ...O o O

# ip route add 127.0.0.0/8 dev lo

A thing about sed regex

2005-05-26T08:57:00.000-07:00

Okay so on the left side you need to escape things like "(" "?" which are needed for example:

To remove - from -bash from ps axo command listing ,i needed to do this.

#ps axo command|sed -ne "1d;s/^-$.*$/\1/p"

And to make that thing optional,dont ask me why i did that ..yeah so to print all commands and to remove - from -bash when encountered.

#ps axo command|sed -ne "1d;s/^-\?$.*$/\1/p"

command command

2005-05-26T08:16:00.000-07:00

# ls
awk complete2.sh complete.sh downloads medrec-1.1 ps.sh rmmod.sh test.sh url.sh
# command ls
awk complete2.sh complete.sh downloads medrec-1.1 ps.sh rmmod.sh test.sh url.sh

Now command executes a command yeah " Run command with args suppressing the normal shell function lookup"

So my deduction is color gotta be a shell function ;)

Basic usage of awk

2005-05-23T22:55:00.000-07:00

I keep forgetting this,so i posted this.This is how you change the the way awk tokenizes...Here I use '/' instead of whitespace

echo "http://www.google.com/test" | awk -F/ '{print $4}'

My remote package installer ;)

2005-05-23T03:43:00.000-07:00

helps u when you dont want to copy urls from the Redhat RPMS directory .
It depends heavily on the links output so it maynot work always..
Didnt work for vedanta... :D

#!/bin/bash
url="http://hostname/AS3u3/RedHat/RPMS/"
package=`links --dump ${url} |grep $1 |awk '{print $3}'`
rpm -ivh ${url}$package

My Package Remover

2005-05-12T01:44:00.000-07:00

It all started with me wanting to clean up a system with zillion packages .
After an half ass attempt to find out what are the packages with biggest size,vedanta pointed me to the rpm's --queryformat option.

So this solved part of the job,finding out what packages to remove.
So here it is .

$rpm --queryformat="%{size} %{name}\n" -qa |sort -n

After this obviously when you want to remove a package ,it takes time as the package maybe required by other packages.So i thought of writing a small bash script to do it.So here goes


#!/bin/sh
removepackage()
{
local fullpackage=$*
if rpm -e $fullpackage 2>/dev/null
 then
    echo "Removed the damn package"
    exit
else
    rpm -e $*  2>&1 | awk -F")" '{print $2}' > /tmp/removepackages
                    while read package

                    do
                    fullpackage="$fullpackage $package"
                    done < /tmp/removepackages
fi 
 echo $fullpackage
 removepackage  $fullpackage 
} 

removepackage $*

So thats it just run it with the a package name as an argument and toast your system.
By and by got to know that if you use a pipeline in the while loop above fullpackage scope wont last after the while loop.Bash will put the value it has before it entered the loop thats because while is done using a subshell.
But how does it matter if its a pipe or a redirection??......

Using ebuild

2005-05-11T01:36:00.000-07:00

Now i wanted to try out d4x on vedanta's (http://vedanta.freecoolsite.com) recommendation so i tried to emerge it.But the compile failed .So i looked up gentoo's bugzilla and found the cause.gtk+ 2.6 has a function with the same name as one used by d4x so there is a conflict.

So i got the patch ( http://bugs.gentoo.org/attachment.cgi?id=49573 ) from the bugzilla and followed these steps to install the package.

$ebuild /usr/portage/net-misc/d4x/d4x-2.5.0.ebuild fetch
$ebuild /usr/portage/net-misc/d4x/d4x-2.5.0.ebuild unpack
$cd /var/tmp/portage/d4x-2.5.0/work/d4x-2.5.0final/
$patch -p0 < /tmp/d4x.patch $ebuild /usr/portage/net-misc/d4x/d4x-2.5.0.ebuild compile $ebuild /usr/portage/net-misc/d4x/d4x-2.5.0.ebuild install $ebuild /usr/portage/net-misc/d4x/d4x-2.5.0.ebuild qmerge

That's it. Enjoy :)

Silencing grep

2005-05-10T05:07:00.000-07:00

$ ps aux|grep xinit
kalyan 28918 0.0 0.0 1484 472 pts/2 R+ 17:37 0:00 grep xinit
$ ps aux|grep -q xinit

the -q option to grep is useful when you dont want to see the output in a script but want to just test for a match using the return value.

if $( ps aux|grep xinit); then echo "yo"; fi

Now this will try to run the output of that command $( ps aux|grep xinit)
and bash will say

-bash: kalyan: command not found

but now when you write a for loop when you have to use it else bash will complain.
So the right thing to do is

$ if ps aux|grep -q xinit; then echo "yo"; fi

$for mycom in $(ps aux|grep xinit); do echo "yo"; done

I know the examples are weird ;)

A thing about nfs

2005-05-10T03:04:00.000-07:00

Though I thought i should only blog about bash related stuff,i like breaking the rules sometimes.
In case you want a read write mount then you have to set up a few things at server end , basically not to map the root on the client to anonymous on the server.

Here is the thing you'd like to add to /etc/exports

/storage *(rw,no_root_squash,sync)

Finding use

2005-05-08T22:51:00.000-07:00

Finding use for using find

$ find -size +100000k
./.evolution/mail/local/Inbox
./fun/kernel/src/linux-2.6.11-gentoo-r4/cscope.out
./fun/kernel/src/linux-2.6.11.7/cscope.out
./eclipse.tgz

$ find -size +100000k -and -name 'In*'
./.evolution/mail/local/Inbox

$ find -size +100000k ! -name 'In*'
./fun/kernel/src/linux-2.6.11-gentoo-r4/cscope.out
./fun/kernel/src/linux-2.6.11.7/cscope.out

$ find -size +100000k -or -name 'cscope_maps.vim*'
./.evolution/mail/local/Inbox
./fun/kernel/src/linux-2.6.11-gentoo-r4/cscope.out
./fun/kernel/src/linux-2.6.11.7/cscope.out
./.vim/plugin/cscope_maps.vim

$ find -amin -1 -name '*.c'
./test.c

$ find . -name "test.c" -and -maxdepth 1 -exec ls -l '{}' \;

ulimit

2005-05-05T01:45:00.000-07:00

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7168
virtual memory (kbytes, -v) unlimited

For core file generation

$ulimit -c unlimited

Named pipes

2005-05-05T01:34:00.000-07:00

$mkfifo pipe

On terminal 1

$cat pipe

On terminal 2

$cat > pipe

Now you know what i mean

Using file test - Part 2

2005-05-04T00:58:00.000-07:00

if [ -p "/dev/initctl" ]
then
echo "I am the pipe file"
fi

if [ -b "/dev/hda" ]
then
echo "I am the block file"
fi

if [ -c "/dev/vc/1" ]
then
echo "I am the character file"
fi

Using file test - Part 1

2005-05-03T23:49:00.000-07:00

#!/bin/bash
#filetest.sh

if [ -x "filetest.sh" ]
then
echo "filetest.sh is executable"
else
echo "filetest.sh is not executable"
fi

$ sh filetest.sh
filetest.sh is not executable
$ chmod +x filetest.sh
$ ./filetest.sh
filetest.sh is executable

Now time for something interesting ..let me find out the executable files inside my home directory using this.Well i dont like the find options.

cat filetest.sh
#!/bin/bash

if [ -d "$1" ]
then
:
else
echo "A directory expected"
exit 1;
fi

for myfile in $( find $1 )
do
if [ -x "$myfile" -a -f "$myfile" ]
then
echo "$myfile is executable"
fi
done

A bash webserver

2005-05-03T05:04:00.000-07:00

Man this thing rocks.Check it out
http://ufpr.dl.sourceforge.net/sourceforge/mws/mws10.tgz

Dancing bars

2005-05-03T04:30:00.000-07:00

while true; do if(( count % 2 == 0)); then tput clear; echo '\'; else tput clear ;echo "/"; fi; ((count++)); done

Now i'll use switch case

#!/bin/sh

printbar()
{
case $1 in
'0') echo "/";;
'1') echo "|";;
'2') echo '\';;
'3') echo "-";;
'4') echo "|";;
'5') echo '\';;
'6') echo "-";;
*) echo "not found";;
esac
}
clear
count=0
while true
do
tput cup 10 10
printbar $(($count % 7))
(( count++ ))
done

Constantly monitor disc usage

2005-05-03T03:57:00.000-07:00

while true;do du -sm .;sleep 1;done

The truth about -z and -n

2005-05-02T03:06:00.000-07:00

if [ -z $null ]
then
echo '$null is null'
else
echo '$null file is not null.'
fi

if [ -n $null ]
then
echo '$null file is not null.'
else
echo '$null is null'
fi

The output is
$$null is null
$$null file is not null.

Not quite as we expected,So we learn that we should always quote the tested string and -n needs it.

if [ -z $null ]
then
echo '$null is null'
else
echo '$null file is not null.'
fi

if [ -n "$null" ]
then
echo '$null file is not null.'
else
echo '$null is null'
fi

Already using the bash power :)

2005-04-30T02:46:00.000-07:00

In the DBUS python code,i came across few examples which had a missing interpreter at the top .
So i sent a mail to mailing list and J5 let me contribute by asking me to submit the patch.
I really like the spirit of Free Software ,i started creating the patches after creating a couple of them.I said
"Wait"...Whats the point of bash if you keep repeating the samething. :)
So here is the script that creates the patch for the missing interpreter.

#!/bin/bash

for sourcefile in $(ls *.py )
do
if head -n 1 $sourcefile |grep -e '^#!' &> /dev/null
then
echo "$sourcefile has some interpretor"
else
cp $sourcefile ${sourcefile}_orig
cat ${sourcefile}_orig | sed -e '1i#!/usr/bin/env python\' -e '' > $sourcefile
diff -u ${sourcefile}_orig $sourcefile >> examples.patch
fi
done

Yo

strlen of Bash

2005-04-29T02:15:00.000-07:00

$ echo ${#EDITOR}
9
$ echo $EDITOR
/bin/nano

Yo!

Shell Parameter Expansion

2005-04-28T02:19:00.000-07:00

This one is big so here I go.

${!prefix*}

This one expands the variables whose name starts with the prefix.One thing to note is that  '*" is needed as shown below

$ echo ${!H*}
HISTCMD HISTFILE HISTFILESIZE HISTSIZE HOME HOSTNAME HOSTTYPE
$ echo ${!HISTCMD}

echo ${!HISTCMD*}
HISTCMD

${#parameter}
length in characters of the parameter but in case * and @ they refer to number of parameters.


$ cat parameter.sh
#!/bin/bash

echo ${#1}
echo $(#*}
$./parameter.sh abcdefghijklmnopqrstuvwxyz
26
1

${parameter#word}
${parameter##word}
try to match the word(it can be a pattern) against the parameter at the beginning and if it matches delete the shortest match $ cat parametermatch.sh #!/bin/bash echo ${1#ab*} echo ${1##ab*} $ ./parametermatch.sh abcdefghi cdefghi
$

${parameter%word}
${parameter%%word}
Samething as above but matches and deletes at the end
$ ./parametermatch.sh abcdefghi abcdefg $ cat parametermatch.sh #!/bin/bash echo ${1%*hi} echo ${1%%*hi}

${parameter/pattern/string}

$ cat parametermatch.sh
#!/bin/bash

echo ${1/pattern/replacepattern}
$ ./parametermatch.sh hipattern
hireplacepattern

Bash Expansions

2005-04-28T01:31:00.000-07:00

I'll come back to numbers before i post some cool features of bash.
Note that the '$' at the beginning of the line denotes the prompt.

1.Brace Expansion

$ echo alumn{a,i,ae,us}
alumna alumni alumnae alumnus

2.Tilde Exansion.

$ echo ~
/home/kalyan

3.Command substitution

$ ls $(which ls)
/bin/ls

Backticks also..

$ ls `which ls`
/bin/ls

4.Filename exapansion

$ ls s*
sd shift.sh string.sh stringcmp.sh
$ ls s?
sd
$ ls s[hd]*
sd shift.sh

5.Arithmetic Expansion
$ n=7
$ ((n=n+10))
$ echo $n
17

6.Paramenter expansion

$ ps=${ps:=lambadamda}
$ echo $ps
lambadamda

Bash Comparisons

2005-04-28T00:07:00.000-07:00

So here is the bottomline as far comparing strings and integers in bash is concerned.
You want to compare strings use

[[ $stringa < $stringb ]]

You want to compare numbers use

(( $numbera < $numberb ))

This will take you a long way without much heartburns. :)
Now lets try out a few examples to understand this thing better.

#!/bin/bash
#string.sh
EQUAL="equal"

if [ $EQUAL == "equal" ]
then
echo "I am equal"
else
echo "I am not equal"
fi

$ ./string.sh
I am equal

Now lets try something more now see if 'a' < 'b'
#!/bin/bash
#stringcmp.sh
A="a"

if [ $A < "b" ]
then
echo "a is lesser than b"
else
echo "Something went wrong"
fi

$ ./stringcmp.sh
./stringcmp.sh: line 5: b: No such file or directory
Something went wrong

Ah so we cant do that. :) . So we try '[[ ]]'

#!/bin/bash
#stringcmp.sh
A="a"

if [[ $A < "b" ]]
then
echo "a is lesser than b"
else
echo "Something went wrong"
fi

$ ./stringcmp.sh
a is lesser than b

So a more comprehensive example

#!/bin/bash
#stringcmp.sh
A="a"
C="c"
B="b"

if [[ $A < "b" ]]
then
echo "a is lesser than b"
else
echo "Something went wrong"
fi
if [[ $C < "b" ]]
then
echo "Something went wrong"
else
echo "c is not lesser than b"
fi
if [[ $B < "b" ]]
then echo "Something went wrong"
else
if [[ $B == "b" ]]
then
echo "b is equal to b"
fi
fi

$ ./stringcmp.sh
a is lesser than b
c is not lesser than b
b is equal to b

Numbers are pretty straightforward as they only work with circular parenthesis "(())".
So you cant use the examples listed above to compare numbers.