Command Center

Step 1:
#end with bb or cc

grep "bb$" virtusertable

Step 2:
#print column1 and colum2 and sort by column2 and keep the first unique line

awk '{print $1,$2}' virtusertable | sort -k2 -u > sortcolum2u.txt

Step 3:
#That will delete lines containing crap[0-3].

grep -Ev 'crap0|crap1|crap2|crap3'

Step 4:
#keep first column

awk '{print $1}' virtusertable 

Step 5:
#print column1 and column2, sort by column2(domain)

awk -F@ '{print $1,$2}' virtusertableColumn1.txt|sort -k2 > virtusertableFsortdomain.txt

Step 6:
##replace space with @

:%s/\s/@/g

REFERENCES
http://www.linuxquestions.org/questions/linux-general-1/sed-or-grep-delete-lines-containing-matching-text-446640/

http://efreedom.com/Question/1-1915636/Way-Uniq-Column
http://stackoverflow.com/questions/2978361/uniq-in-awk-removing-duplicate-values-in-a-column-using-awk
http://www.softpanorama.org/Tools/sort.shtml



Counting unique values in a column with a shell script

$ cut -f2 file.txt | sort | uniq | wc -l

#Finding unique records from file
01224624005
01224626366
01224627408
01224626366
01224627408

##return duplicated lines 01224626366 01224627408
uniq -d sorted_sme.txt > dup_sme.txt
##return unique lines  01224624005
uniq -u sorted_sme.txt > unq_sme.txt

REFERENCES
http://www.computing.net/answers/unix/finding-unique-records-from-file/4591.html

Processing the delimited files using cut

cut command print selected parts of lines from each FILE (or variable) i.e. it remove sections from each line of files:

For example /etc/passwd file is separated using character : delimiters.

To print list of all users, type the following command at shell prompt:

$ cut -d: -f1 /etc/passwd

Output:

root
you
me
vivek
httpd

Where,

* -d : Specifies to use character : as delimiter
* -f1 : Print first field, if you want print second field use -f2 and so on...

Now consider variable service. Let us print out mail word using cut command:
$ service="http mail ssh"
$ echo $service | cut -d' ' -f2

mail

Note that a blank space is used as delimiter.
Processing the delimited files using awk

You can also use awk command for same purpose:

$ awk -F':' '{ print $1 }' /etc/passwd

Output:

root
you
me
vivek
httpd

Where,

* -F: - Use : as fs (delimiter) for the input field separator
* print $1 - Print first field, if you want print second field use $2 and so on


REFERENCES
http://www.cyberciti.biz/tips/processing-the-delimited-files-using-cut-and-awk.html

have an ascii list of about 13,000 opt-in email addresses.

Each email address is on a separate line.

I need a browser based perl script that will help me extract the aol, hotmail and yahoo emails from that text file and separate them into 4 text files residing on the server for later downloading.

The emails are in total random order and are like the following

rowby@aol.com
rowby@earthlink.com
rowby@rowby.com
rowby@hotmail.com
rowby@yahoo.com
etc
etc
etc


I want to be able to upload the text file to the server, and then have the perl script process it. It will create an AOL list, a Yahoo List, A Hotmail list -- and an "Other" list.

I can use FTP to download those text files -- unless you have something more elegant.

Just to add some more "spice" to this request, I would like the list to be created in the following format:

A|john@sampledomain.com|-|-|-|01|01|2001
A|mary@otherdomain.com|-|-|-|01|01|2001
etc

The ONLY thing that will change on each line is the email address.

THanks

Rowby



Solution:
rowby:
Hi all,

Maneshr worked all saturday morning on the server and got the script working and I'm awarding him the points.

However, you have all contributed to this educational process and my client has authorized 2 free day car rentals to all who have submitted suggestions and scripts to this question (certain restrictions apply).

The client has offices in California only at this time, and you can visit the site at www.foxrentacar.com. If you are in California please email me at rowby@foxrentacar.com and I'll set it up.

By the way this script is for a newsletter that has a New YEars day theme, so I appreciated all of your quick responses -- I will be spending the rest of the weekend using the automated program that this script will work with, carefully sending out (slowly but surely) the newsletters.


Now here is the final script:

#!/usr/bin/perl

$|++;     ##     Disable output buffering

##  Print the MIME header
print "Content-type: text/html\n\n";

##     Print the HTML title.
print "<TITLE>Rowby's Email splitting page</TITLE>\n";

##  Open the data file for reading.
open(IN,"/home/sites/site4/web/cgi-bin/row.txt") || die $!;
@lines=<IN>;  ##  Read each line of the file as an element of the array
close(IN);

##  Remove the newline character from the end of each element.
chomp(@lines);

##  Process each element of the array.
foreach $email (@lines){
     ##  Extract just the domain name from the email id.
     $email=~ /\@(.*)/;  ##  Get the ENTIRE domain name.
     ($domain)=split(/\./,$1); ##  Extract ONLY the domain name.
     $domain=lc($domain);     ##     Convert that extracted domain to lowercase.

     next if (!($domain) || $domain=~ /^\s+$/);     ##     Ignore empty domains

      ##  Change the if statement below to add another domain name.
      if ($domain!~ /(aol)|(hotmail)|(yahoo)/i){  ##  Domain name is not from our list of special ones. Others!!
           push(@others,'A|'.$email.'|-|-|-|01|01|2001'); ##  Store this email id in a common array.
           $files{'others'}++;   ##  Increment the count of these common email domains.
      }else{  ##  This domain name is a special one.
           push(@$domain,'A|'.$email.'|-|-|-|01|01|2001');  ##  Store is seperately, in its unique array. E.g. all hotmail.com ids go in @hotmail array.
           $files{$domain}++;    ##  Increment the count of these special email domains.
     }
}

##  Process each type of email domain (viz. special ones, like hotmail, aol etc.. & common ones i.e. others)
foreach (sort keys %files){
     $total+=scalar(@$_);
     print "Email ids for domain $_ = ",scalar(@$_),"<BR>\n";

     ##create an ouput file with the same name as domain name.
     ##       E.g. all hotmail.com ids will be stored in hotmail.txt
     $outfile='/home/sites/site4/web/cgi-bin/'.$_.'.txt';
     open(OUT,">$outfile") || die $!;
     ##  Now that all sorting has been done, write the data to the proper output files.
     print OUT join("\n",@$_)."\n";
     close (OUT);
}

print "<P>Grant total of email ids = $total<BR>\n";

How Can I Sort Linux Files?
The sort command sorts a file according to fields--the individual pieces of data on each line. By default, sort assumes that the fields are just words separated by blanks, but you can specify an alternative field delimiter if you want (such as commas or colons). Output from sort is printed to the screen, unless you redirect it to a file.
If you had a file like the one shown here containing information on people who contributed to your presidential reelection campaign, for example, you might want to sort it by last name, donation amount, or location. (Using a text editor, enter those three lines into a file and save it with donor.data as the file name.)

Bay Ching 500000 China
Jack Arta 250000 Indonesia
Cruella Lumper 725000 Malaysia

Let's take this sample donors file and sort it according to the donation amount. The following shows the command to sort the file on the second field (last name) and the output from the command:

sort +1 -2 donors.data
Jack Arta 250000 Indonesia
Bay Ching 500000 China
Cruella Lumper 725000 Malaysia

The syntax of the sort command is pretty strange, but if you study the following examples, you should be able to adapt one of them for your own use. The general form of the sort command is

sort

The most common flags are as follows:

-f Make all lines uppercase before sorting (so "Bill" and "bill" are treated the same).
-r Sort in reverse order (so "Z" starts the list instead of "A").
-n Sort a column in numerical order
-tx Use x as the field delimiter (replace x with a comma or other character).
-u Suppress all but one line in each set of lines with equal sort fields (so if you sort on a field containing last names, only one "Smith" will appear even if there are several).

Specify the sort keys like this:

+m Start at the first character of the m+1th field.
-n End at the last character of the nth field (if -N omitted, assume the end of the line).

Looks weird, huh? Let's look at a few more examples with the sample company.data file shown here, and you'll get the hang of it. (Each line of the file contains four fields: first name, last name, serial number, and department name.)

Jan Itorre 406378 Sales
Jim Nasium 031762 Marketing
Mel Ancholie 636496 Research
Ed Jucacion 396082 Sales

To sort the file on the third field (serial number) in reverse order and save the results in sorted.data, use this command:

sort -r +2 -3 company.data > sorted.data
Mel Ancholie 636496 Research
Jan Itorre 406378 Sales
Ed Jucacion 396082 Sales
Jim Nasium 031762 Marketing

Now let's look at a situation where the fields are separated by colons instead of spaces. In this case, we will use the -t: flag to tell the sort command how to find the fields on each line. Let's start with this file:

Itorre, Jan:406378:Sales
Nasium, Jim:031762:Marketing
Ancholie, Mel:636496:Research
Jucacion, Ed:396082:Sales

To sort the file on the second field (serial number), use this command:

sort -t: +1 -2 company.data
Nasium, Jim:031762:Marketing
Jucacion, Ed:396082:Sales
Itorre, Jan:406378:Sales
Ancholie, Mel:636496:Research

To sort the file on the third field (department name) and suppress the duplicates, use this command:

sort -t: -u +2 company.data
Nasium, Jim:031762:Marketing
Ancholie, Mel:636496:Research
Itorre, Jan:406378:Sales

Note that the line for Ed Jucacion did not print, because he's in Sales, and we asked the command (with the -u flag) to suppress lines that were the same in the sort field.

There are lots of fancy (and a few obscure) things you can do with the sort command. If you need to do any sorting that's not quite as straightforward as these examples, try the man sort command for more information.

For more information on the sort command, see the sort manual.

Previous Lesson: Heads or Tails?
Next Lesson: Eliminating Duplicates

Question: I've downloaded configuration file in a PDF format. I do not have GUI installed on remote Linux / UNIX server. How do I convert a PDF (Portable Document Format) file to a text format using command line so that I can view file over remote ssh session?

Answer: Use pdftotext utility to convert Portable Document Format (PDF) files to plain text. It reads the PDF file, and writes a text file. If text file is not specified, pdftotext converts file.pdf to file.txt. If text-file is -, the text is sent to stdout.

Install pdftotext under RedHat / RHEL / Fedora / CentOS Linux

pdftotext is installed using poppler-utils package under various Linux distributions:
# yum install poppler-utils
OR use the following under Debian / Ubuntu Linux
$ sudo apt-get install poppler-utils

pdftotext syntax

pdftotext {PDF-file} {text-file}

How do I convert a pdf to text?

Convert a pdf file called hp-manual.pdf to hp-manual.txt, enter:
$ pdftotext hp-manual.pdf hp-manual.txt

Specifies the first page 5 and last page 10 (select 5 to 10 pages) to convert, enter:
$ pdftotext -f 5 -l 10 hp-manual.pdf hp-manual.txt

Convert a pdf file protected and encrypted by owner password:
$ pdftotext -opw 'password' hp-manual.pdf hp-manual.txt

Convert a pdf file protected and encrypted by user password:
$ pdftotext -upw 'password' hp-manual.pdf hp-manual.txt

Sets the end-of-line convention to use for text output. You can set it to unix, dos or mac. For UNIX / Linux oses, enter:
$ pdftotext -eol unix hp-manual.pdf hp-manual.txt

Further readings:

• man page pdftotext

REFERENCES
http://www.cyberciti.biz/faq/converter-pdf-files-to-text-format-command/

How do I use awk pattern scanning and processing language under bash scripts? Can you provide a few examples?

Awk is an excellent tool for building UNIX/Linux shell scripts. AWK is a programming language that is designed for processing text-based data, either in files or data streams, or using shell pipes. In other words you can combine awk with shell scripts or directly use at a shell prompt.
Print a Text File

awk '{ print }' /etc/passwd
OR
awk '{ print $0 }' /etc/passwd
Print Specific Field

Use : as the input field separator and print first field only i.e. usernames (will print the the first field. all other fields are ignored):
awk -F':' '{ print $1 }' /etc/passwd
Send output to sort command using a shell pipe:
awk -F':' '{ print $1 }' /etc/passwd | sort
Pattern Matching

You can only print line of the file if pattern matched. For e.g. display all lines from Apache log file if HTTP error code is 500 (9th field logs status error code for each http request):
awk '$9 == 500 { print $0}' /var/log/httpd/access.log
The part outside the curly braces is called the "pattern", and the part inside is the "action". The comparison operators include the ones from C:

== != < > <= >= ?:

If no pattern is given, then the action applies to all lines. If no action is given, then the entire line is printed. If "print" is used all by itself, the entire line is printed. Thus, the following are equivalent:
awk '$9 == 500 ' /var/log/httpd/access.log
awk '$9 == 500 {print} ' /var/log/httpd/access.log
awk '$9 == 500 {print $0} ' /var/log/httpd/access.log
Print Lines Containing tom, jerry AND vivek

Print pattern possibly on separate lines:
awk '/tom|jerry|vivek/' /etc/passwd
Print 1st Line From File

awk "NR==1{print;exit}" /etc/resolv.conf
awk "NR==$line{print;exit}" /etc/resolv.conf
Simply Arithmetic

You get the sum of all the numbers in a column:
awk '{total += $1} END {print total}' earnings.txt
Shell cannot calculate with floating point numbers, but awk can:
awk 'BEGIN {printf "%.3f\n", 2005.50 / 3}'
Call AWK From Shell Script

A shell script to list all IP addresses that accessing your website. This script use awk for processing log file and verification is done using shell script commands.

#!/bin/bash
d=$1
OUT=/tmp/spam.ip.$$
HTTPDLOG="/www/$d/var/log/httpd/access.log"
[ $# -eq 0 ] && { echo "Usage: $0 domain-name"; exit 999; }
if [ -f $HTTPDLOG ];
then
awk '{print}' $HTTPDLOG >$OUT
awk '{ print $1}' $OUT | sort -n | uniq -c | sort -n
else
echo "$HTTPDLOG not found. Make sure domain exists and setup correctly."
fi
/bin/rm -f $OUT

AWK and Shell Functions

Here is another example. chrootCpSupportFiles() find out the shared libraries required by each program (such as perl / php-cgi) or shared library specified on the command line and copy them to destination. This code calls awk to print selected fields from the ldd output:


chrootCpSupportFiles() {
# Set CHROOT directory name
local BASE="$1" # JAIL ROOT
local pFILE="$2" # copy bin file libs

[ ! -d $BASE ] && mkdir -p $BASE || :

FILES="$(ldd $pFILE | awk '{ print $3 }' |egrep -v ^'\(')"
for i in $FILES
do
dcc="$(dirname $i)"
[ ! -d $BASE$dcc ] && mkdir -p $BASE$dcc || :
/bin/cp $i $BASE$dcc
done

sldl="$(ldd $pFILE | grep 'ld-linux' | awk '{ print $1}')"
sldlsubdir="$(dirname $sldl)"
if [ ! -f $BASE$sldl ];
then
/bin/cp $sldl $BASE$sldlsubdir
else
:
fi
}

This function can be called as follows:
chrootCpSupportFiles /lighttpd-jail /usr/local/bin/php-cgi
AWK and Shell Pipes

List your top 10 favorite commands:
history | awk '{print $2}' | sort | uniq -c | sort -rn | head
Sample Output:

172 ls
144 cd
69 vi
62 grep
41 dsu
36 yum
29 tail
28 netstat
21 mysql
20 cat

whois cyberciti.com | awk '/Domain Expiration Date:/ { print $6"-"$5"-"$9 }'
Awk Program File

You can put all awk commands in a file and call the same from a shell script using the following syntax:
awk -f mypgoram.awk input.txt
Awk in Shell Scripts - Passing Shell Variables TO Awk

You can pass shell variables to awk using the -v option:


n1=5
n2=10
echo | awk -v x=$n1 -v y=$n2 -f program.awk

Assign the value n1 to the variable x, before execution of the program begins. Such variable values are available to the BEGIN block of an AWK program:

BEGIN{ans=x+y}
{print ans}
END{}




some of my one liner awk tricks:

— To convert squid log timestamps to readable, sortable format:

gawk '{print strftime("%m/%d %H:%M:%S ",$1)" "substr($0,12,999)}' access.log > dated

— To avoid having to cut/paste the above, I have in my .profile file:

alias tim="gawk '{print strftime(\"%m/%d %H:%M:%S \",\$1)\" \"substr(\$0,12,999)}'"

— To use AWK to process comma separated data:

awk -F, '{print $1, "," $6}' excel-save.csv> extract.csv

— To count complex pattern occurrence

awk '{if (substr($2,1,4) == "2008" && $4 == "Exception") {print $1}}' test|grep -c

1. import .pst into outlook
2. export the bounce folder to .excel
3. extract the bounce From address into linux

grep -C 2 "fatal errors" nurseaug3.txt > nurseaug3a.txt

perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' emails.txt | sort -u > output.txt

find . -name "*.txt" | xargs perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' emails.txt | sort -u > output.txt

Also see:
Reference:
1.http://lifehacker.com/391205/email-address-extract-grabs-addresses-from-any-file



2. Yeah, that's just a perl script wich takes in a file and checks every word to see if it's a valid email address. It prints it out if so. Here's a commented version:

#!/usr/local/bin/perl -w
use strict;
# that stuff is just to make it a perl script

# email::Valid is a module to check for valid email addresses
# you can get it from CPAN.org along with tons of other modules
# If you're using perl on windows, i bet activestate has a version.
# The author says that it may be slow on Win32 if you have addresses
# where there is no nameserver to check them against.
use email::Valid;

# this loops over each line in the input
while (<>) {
# this loops over each "word" in the line (it splits on whitespace)
for my $word ( split() ) {
# if it's a valid address..
if ( my $address = email::Valid->address( $word ) ) {
# print it out.
print $address, "\n";
}
}
}

Put it in a file and call the file "getemails.pl" or something, then send all of your files to it:
./getemails.pl < somefile.txt or cat * ¦ ./getemails.pl and wait for your list of emails to come out. I just tested it and it seems to do pretty well. -Andy

#!/usr/bin/env python
'''
  emailsfromfile.py -- Get all unique email addresses from a file

  by Patrick Mylund Nielsen
  http://patrickmylund.com/projects/emailsfromfile/

  License: WTFPL (http://sam.zoy.org/wtfpl/)
'''

__version__ = '1.1'

import sys
import os
import re
import codecs

# Regular expression matching according to RFC 2822 (http://tools.ietf.org/html/rfc2822)
rfc2822_re = r"""(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])"""
email_prog = re.compile(rfc2822_re, re.IGNORECASE)

def isEmailAddress(string):
    return email_prog.match(string)

def main(filename, separator='\n', encoding=None):
    separator_replace = {
        'space': ' ',
        'newline': '\n',
    }
    if not os.path.isfile(filename):
        raise IOError("%s is not a file." % filename)
    results = set()
    with codecs.open(filename, 'rb', encoding) as f:
        for line in f:
            results.update(email_prog.findall(line))
    for k, v in separator_replace.iteritems():
        separator = separator.replace(k, v)
    print(separator.join(results))

if __name__ == '__main__':
    args = len(sys.argv) - 1
    if 0 < args < 4:
        main(*sys.argv[1:])
    else:
        print("Usage: python %s <filename> [separator] [encoding]" % sys.argv[0])
        print("The default separator is a newline. To separate by space, literally enter 'space' as the separator.")

Usage

python emailsfromfile.py [separator] [encoding]

The separator and encoding parameters are optional. The separator is a new line and the file encoding is 8-bit ASCII by default. If you want to specify an encoding, you also have to set a separator; to use a new line (the default), specify newline as the separator.

Examples:

python emailsfromfile.py contacts.csv — returns all email addresses from contacts.csv, displaying one email address per line
python emailsfromfile.py contacts.csv , — returns a comma-separated list of all email addresses in contacts.csv
python emailsfromfile.py contacts.csv space — returns all email addresses from contacts.csv, separated by a space
python emailsfromfile.py contacts.csv ; > emails.txt — writes all of the email addresses from contacts.csv, separated by a semi-colon, to emails.txt
python emailsfromfile.py utf8-contacts.csv newline utf-8 — returns all email addresses, one per line, from the UTF-8 encoded file utf8-contacts.csv

References:
http://www.webmasterworld.com/forum10/1195.htm
http://patrickmylund.com/projects/emailsfromfile/