Friday, April 1, 2011

Make Browsers Cache Static Files With mod_expires On Apache2 (Debian Squeeze)

SkyHi @ Friday, April 01, 2011
This tutorial explains how you can configure Apache2 to set the Expires HTTP header and the max-age directive of the Cache-Control HTTP header of static files (such as images, CSS and Javascript files) to a date in the future so that these files will be cached by your visitors' browsers. This saves bandwidth and makes your web site appear faster (if a user visits your site for a second time, static files will be fetched from the browser cache). This tutorial was written for Debian Squeeze.

I do not issue any guarantee that this will work for you!


1 Preliminary Note

I'm assuming you have a working Apache setup on your Debian Squeeze server, e.g. as shown in this tutorial: Installing Apache2 With PHP5 And MySQL Support On Debian Squeeze (LAMP)


2 Enabling mod_expires

mod_expires can be enabled as follows:

a2enmod expires

Restart Apache afterwards:

/etc/init.d/apache2 restart


3 Configuring mod_expires

The mod_expires configuration can be placed in the overall Apache server configuration, inside a virtual host container, inside a directive, or inside an .htaccess file.

In this example, I will place it in Apache's default vhost which is configured in /etc/apache2/sites-available/default on Debian Squeeze:

vi /etc/apache2/sites-available/default

If you have multiple file types that should expire after the same time after they have been accessed (let's say in one week), you can use a combination of the FilesMatch and the ExpiresDefault directives, e.g. as follows:

[...]
<IfModule mod_expires.c>
          <FilesMatch "\.(jpe?g|png|gif|js|css)$">
                      ExpiresActive On
                      ExpiresDefault "access plus 1 week"
          </FilesMatch>
</IfModule>
[...]
This would tell browsers to cache .jpg, .jpeg, .png, .gif, .js, and .css files for one week.

Restart Apache after your changes:

/etc/init.d/apache2 restart

Instead of using FilesMatch and ExpiresDefault directives, you could also use the ExpiresByType directice and set an Expires header (plus the max-age directive of the Cache-Control HTTP header) individually for each file type, e.g. as follows:

[...]
<IfModule mod_expires.c>
          ExpiresActive on

          ExpiresByType image/jpg "access plus 60 days"
          ExpiresByType image/png "access plus 60 days"
          ExpiresByType image/gif "access plus 60 days"
          ExpiresByType image/jpeg "access plus 60 days"

          ExpiresByType text/css "access plus 1 days"

          ExpiresByType image/x-icon "access plus 1 month"

          ExpiresByType application/pdf "access plus 1 month"
          ExpiresByType audio/x-wav "access plus 1 month"
          ExpiresByType audio/mpeg "access plus 1 month"
          ExpiresByType video/mpeg "access plus 1 month"
          ExpiresByType video/mp4 "access plus 1 month"
          ExpiresByType video/quicktime "access plus 1 month"
          ExpiresByType video/x-ms-wmv "access plus 1 month"
          ExpiresByType application/x-shockwave-flash "access 1 month"

          ExpiresByType text/javascript "access plus 1 week"
          ExpiresByType application/x-javascript "access plus 1 week"
          ExpiresByType application/javascript "access plus 1 week"
</IfModule>
[...]

You might have noticed that I've set three ExpiresByType directives for Javascript files - that is because Javascript files might have different file types on each server. If you set just one directive for text/javascript, but the server recognizes the Javascript file as application/javascript, then it will not be covered by your configuration, and no cache headers will be set.

You can use the following time units in your configuration:

years
months
weeks
days
hours
minutes
seconds

Please note that Apache accepts these time units in both singular and plural, so you can use day and days, week and weeks, etc.

It is possible to combine multiple time units, e.g. as follows:

ExpiresByType text/html "access plus 1 month 15 days 2 hours"

Also note that if you use a far future Expires header you have to change the component's filename whenever the component changes. Therefore it's a good idea to version your files. For example, if you have a file javascript.js and want to modify it, you should add a version number to the file name of the modified file (e.g. javascript-1.1.js) so that browsers have to download it. If you don't change the file name, browsers will load the (old) file from their cache.

Instead of basing the Expires header on the access time of the browser (e.g. ExpiresByType image/jpg "access plus 60 days"), you can also base it on the modification date of a file (please note that this works only for real files that are stored on the hard drive!) by using the modification keyword instead of access:

ExpiresByType image/gif "modification plus 7 days"


4 Testing

To test if your configuration works, you can install the Live HTTP Headers plugin for Firefox and access a static file through Firefox (e.g. an image). In the Live HTTP Headers output, you should now see an Expires header and a Cache-Control header with a max-age directive (max-age contains a value in seconds, for example 604800 is one week in the future):

REFERENCES
http://www.howtoforge.com/make-browsers-cache-static-files-with-mod_expires-on-apache2-debian-squeeze

Vim Move the cursor forward and backward to the next occurance of the character

SkyHi @ Friday, April 01, 2011
Efficient Editing With vim

Translations of this article are available in German (Effiziente Textbearbeitung mit Vim), French (L'édition efficace avec vim), Italian (Vim e Gvim: usarli in modo efficiente, and Chinese (高效使用vim).


"To me, vi is Zen.
To use vi is to practice zen.
Every command is a koan.
Profound to the user,
unintelligible to the uninitiated.
You discover truth every time you use it."
--reddy@lion.austin.com

This tutorial assumes a basic knowledge of vim -- insert mode, command mode, loading and saving files, etc. It is intended to help vi novices develop their skills so that they can use vi efficiently.

In this tutorial, means Ctrl-X -- that is, hold down the Ctrl key and press X. You can get help on most of the commands used here by typing :help command in vim, where command is what you need help on.
Moving efficiently
Stay out of insert mode

In general, you want to spend as little of your time in vim's insert mode as possible, because in that mode it acts like a dumb editor. This is why most vim novices spend so much time in insert mode -- it makes vim easy to use. But vim's real power lies in command mode! You'll find that the better you know vim, the less time you will spend in insert mode.
Use h, j, k, and l

The first step to efficient editing in vim is to wean yourself from the arrow keys. One of the the advantages of vim's modal design is that you do not need to constantly move your hands back and forth between the arrow keys and the letter keys; when you are in command mode, the letters h, j, k and l correspond to the directions left, down, up, and right, respectively. It takes some practice to get used to, but you will notice the speed difference once you're used to it.

When you are editing e-mail or other paragraph-formatted text, you might notice that the direction keys skip more lines than you expect. This is because your paragraphs appear as one long line to vim. Type g before h, j, k or l to move by screen lines instead of virtual lines.
Use motions to move the cursor in the current line

Most editors have only simple commands for moving the cursor (left, up, right, down, to beginning/end of line, etc). vim has very advanced commands for moving the cursor; these commands are referred to as motions. When the cursor moves from one point in the text to another, the text between the points (and including the points themselves) is considered to be "moved over." (This will be important later.)

Here are a few of the more useful motions:

fx  Move the cursor forward to the next occurance of the character x on the current line (obviously, x can be any character you like). This is an extremely useful command. You can type ; to repeat the last f command you gave.
tx  Same as above, but moves the cursor to right before the character, not all the way to it. (It's very useful, really.)
Fx  Move the cursor backward to the next occurance of the character x on the current line.
w  Move the cursor forward by a word.
b  Move the cursor backward by a word.
0  Move the cursor to the beginning of the current line.
^  Move the cursor to the first character on the current line.
$  Move the cursor to the end of the line
)  Move the cursor forward to the next sentence. (Useful when editing e-mail or text documents.)
(  Move the cursor backward by a sentence.

Move efficiently through the file

vim has many commands that can send you to where you want to go in your file -- there's rarely a need to scroll manually through it. The below keystrokes are not technically motions, since they move around in the file instead of in a particular line.
  Move the cursor forward by a screenful of text
  Move the cursor backward by a screenful of text
G  Move the cursor to the end of the file
numG  Move the cursor line num. (For instance, 10G moves to line 10.)
gg  Move the cursor to the beginning of the file
H  Move the cursor to the top of the screen.
M  Move the cursor to the middle of the screen.
L  Move the cursor to the bottom of the screen.
*  Read the string under the cursor and go to the next place it appears. (For instance, if your cursor was somewhere on the word "bob," the cursor would move to the next occurance of "bob" in your file.)
#  Same as above, except it moves the cursor to the previous occurance.
/text  Starting from the cursor, find the next occurance of the string text and go to it. You will need to press Enter to execute the search. To re-execute your last search, type n (for next occurance).
?text  Same as /, but searches in the opposite direction.
ma  Make a bookmark named a at the current cursor position. A bookmark can be named any lowercase letter. You can't see the bookmark, but it's there!
`a  Go to bookmark a. Important: that's a backtick, not a single quote. The backtick is located to the left of the 1 on most keyboards.
`.  Go to the line that you last edited. This is very useful! If you need to scroll through the file to look something up, you can go back to where you were without bookmarking it by using the `. command.


Typing efficiently
Use keyword completion

vim has a very nice keyword completion system. This means that you can type part of a long word, press a key, and have vim finish the word for you. For instance, if you have a variable called iAmALongAndAwkwardVarName somewhere in your code, you probably don't want to type the whole thing in every time you use it.

To use keyword completion, just type the first few letters of the string (e.g. iAmAL) and press (that means hold down Ctrl and type N) or . If vim doesn't give you the word you want at first, keep trying -- vim will cycle through all completions it can find.
Enter insert mode intelligently

Most users new to vim get into insert mode by typing i. This works, but it's often pretty inefficient, since vi has a host of commands that leave the editor in insert mode. Here are some of the more popular ones:

i  Insert text to the left of the current character.
I  Insert text at the beginning of the current line.
a  Insert text to the right of the current character.
A  Insert text at the end of the current line.
o  Create a new line under the current one and insert text there.
O  Create a new line above the current one and insert text there.
c{motion}  Delete (change) the text moved over by {motion} and insert text to replace it. For instance, c$ would delete the text from the cursor to the end of the line and enter insert mode. ct! would delete the text from the cursor up to (but not including) the next exclamation mark and enter insert mode. The deleted text is copied to the clipboard and can be pasted.
d{motion}  Delete the text moved over by {motion} -- same as c{motion}, but doesn't enter insert mode.


Moving blocks of text efficiently
Use visual selections and the appropriate selection mode

Unlike the original vi, vim allows you to highlight text and perform operations on it. There are three main visual selection modes (that is, text highlighting modes). These modes are as follows:
v  Characterwise selection mode. This is the selection mode that most people are used to, so practice with it before trying the others.
V  Linewise selection mode. Whole lines are always selected. This is better than characterwise mode when you want to copy or move a group of lines.
  Blockwise selection mode. Extremely powerful and available in very few other editors. You can select a rectangular block and any text inside that block will be highlighted.

All the usual cusor movement keys apply -- so, for instance, vwww would go into visual selection mode and highlight the next three words. Vjj would go into linewise visual selection mode and highlight the current line and the two lines below it.
Cutting and copying from visual selections

Once you have a highlighted selection, you probably want to do something with it. Some of the more useful commands you can give when an area of text is highlighted:
d  Cut (delete) the highlighted text and put it into the clipboard.
y  Copy (or yank, which is vim-ese for "copy") the highlighted text into the clipboard.
c  Cut the highlighted text into the clipboard. This is just like d, except it leaves the editor in insert mode.

Cutting and copying from non-visual selections

If you know exactly what you want to copy or cut, you can do it without entering visual mode. This saves time.
d{motion}  Cut the text moved over by {motion} to the clipboard. For instance, dw would cut a word and dfS would cut from the cursor up to and including the next capital S on the current line of text.
y{motion}  Copy the text moved over by {motion}.
c{motion}  Cut the text moved over by {motion} and leave the editor in insert mode.
dd  Cut the current line.
yy  Copy the current line.
cc  Cut the current line and leave the editor in insert mode.
D  Cut from the cursor to the end of the current line.
Y  Yank the whole line, just like yy. (Yes, it's inconsistent! You can use y$ to do what you would expect Y to do.)
C  Cut from the cursor to the end of the current line and leave the editor in insert mode.
x  Cut the current character. (This is sort of like a command-mode backspace.)
s  Cut the current character and leave the editor in insert mode.


Pasting

Pasting is easy. Put the cursor where you want the pasted text and type p.
Using multiple clipboards

Most editors have a single clipboard. vim has many more; clipboards in vim are called registers. You can list all the currently defined registers and their contents by typing :reg. Typically, you'll be using the lowercase letter registers; the others are used for various internal vim purposes and are only occasionally helpful.

To use a specific register for a copy or paste operation, simply type "a before the command for the operation, where a is the register you want to use.

For example, to copy the current line into register k, you could type "kyy. (You could also type V"ky. Why would that work?). That line would stay in register k until you specifically copied something else into register k. You would then use "kp to paste the text from register k.
Avoiding repetition
The amazing . command

In vi, typing . (a period) will repeat the last command you gave. For instance, if your last command was dw (delete word), vi will delete another word.
Using counts

Counts are one of the most powerful and time-saving features of vim. Any command can be preceded by a number. The number will tell vim how many times to execute the command. Here are a few examples:

3j will move the cursor down three lines.

10dd will delete ten lines.

y3f" will yank (copy) text from the cursor to the third quotation mark after the cursor on the current line. Counts are useful to extend the range of a motion in this manner.
Recording macros

Occasionally, you'll find yourself doing the same thing over and over to blocks of text in your document. vim will let you record an ad-hoc macro to perform the operation.
qregister  Start macro recording into the named register. For instance, qa starts recording and puts the macro into register a.
q  End recording.
@register  Replay the macro stored in the named register. For instance, @a replays the macro in register a.


Keep in mind that macros just record your keystrokes and play them back; they are not magic. Recording macros is almost an art form because there are so many commands that accomplish a given task in vim, and you must carefully select the commands you use while your macro is recording so that they will work in all the places you plan to execute the macro.
Writing code in vim

vim is an excellent editor for source code because it has many features that are specifically designed to help programmers. Here are a few of the more handy ones:
]p  Just like p, but it automatically adjusts the indent level of the pasted code to match that of the code you paste into. Try it!
%  Putting the cursor on a brace, bracket, or parenthesis and pressing % will send the cursor to the matching brace, bracket, or parenthese. Great for fixing parse problems related to heavily nested blocks of code or logic.
>>  Indent the highlighted code. (See the earlier section about efficient text selection. If no text is selected, the current line is indented.)
<<  Like >>, but un-indents.
gd  Go to the definition (or declaration) of the function or variable under the cursor.
K  Go to the man page for the word currently under the cursor. (For instance, if your cursor is currently over the word sleep, you will see the man page for sleep displayed.)


REFERENCES

http://jmcpherson.org/editing.html

find files bigger than certain size and scp over to another server

SkyHi @ Friday, April 01, 2011
Files can be found under Linux in many different ways. Using the find tool is one of the best ways to find files. The find tool has a huge number of parameters which can be set so that Linux finds exactly those files that you were searching for. Many users use the find tool with just the basic parameters. They get the results that they were looking for. Unfortunately most of the users don't spend time to learn all about find. If they do, they can make excellent use of this tool and I am sure you would be surprised at the possibilities.

In case you just want to know where a particular file exists on your system, and nothing else is required, then use locate tool. Article No.20 explains how to use locate.


Here are a few ways to use find

-

$ find / -name 'program.c' 2>/dev/null
$ find / -name 'program.c' 2>errors.txt
/
Start searching from the root directory (i.e / directory)
-name
Given search text is the filename rather than any other attribute of a file
'program.c'
Search text that we have entered. Always enclose the filename in single quotes.. why to do this is complex.. so simply do so.

Note : 2>/dev/null is not related to find tool as such. 2 indicates the error stream in Linux, and /dev/null is the device where anything you send simply disappears. So 2>/dev/null in this case means that while finding for the files, in case any error messages pop up simply send them to /dev/null i.e. simply discard all error messages.

Alternatively you could use 2>error.txt where after the search is completed you would have a file named error.txt in the current directory with all the error messages in it.

-

$ find /home/david -name 'index*'
$ find /home/david -iname 'index*'
The 1st command would find files having the letters index as the beginning of the file name. The search would be started in the directory /home/david and carry on within that directory and its subdirectories only.
The 2nd command would search for the same, but the case of the filename wouldn't be considered. So all files starting with any combination of letters in upper and lower case such as INDEX or indEX or index would be returned.

-

$ find -name met*
The above command would start searching for the files that begin with the letters 'met' within the current directory and the directories that are present within the current directory. Since the directory is not specified as the the second parameter, Linux defaults to using the current directory as the one to start the search in.

-

$ find /mp3collection -name '*.mp3' -size -5000k
$ find / -size +10000k
The 1st command would find within a directory called /mp3collection, only those mp3 files that have a size less than 5000 Kilobytes ( < 5MB) The 2nd command would search from the / directory for any file that is larger than 10000k (> 10MB)

-

$ find /home/david -amin -10 -name '*.c'
$ find /home/david -atime -2 -name '*.c'
$ find /home/david -mmin -10 -name '*.c'
$ find /home/david -mtime -2 -name '*.c'

The 1st commmand searches for those files that are present in the directory /home/david and its subdirectoires which end in .c and which have been accessed in the last 10 minutes.
The 2nd command does the same but searches for those files that have been accessed in the last 10 hours.
The 3rd and the 4th commands do the same as the 1st and 2nd commands but they search for modified files rather than accessed files. Only if the contents of the files have been modified, would their names be returned in the search results.

-

$ find / -mount -name 'win*'
This command searches for files starting with the letters 'win' in their filenames. The only difference is that the mounted filesystems would not be searched for this time. This is useful when you have your Windows partitions mounted by default. And a search for 'win' might return many files on those partitions, which you may not be really interested in. This is only one use of -mount parameter.

-

$ find /mp3-collection -name 'Metallica*' -and -size +10000k
$ find /mp3-collection -size +10000k ! -name "Metallica*"
$ find /mp3-collection -name 'Metallica*' -or -size +10000k
Boolean operators such as AND, OR and NOT make find an extremely useful tool.
The 1st command searches within the directory /mp3-collection for files that have their names beginning with 'Metallica' and whose size is greater than 10000 kilobytes (> 10 MB).
The 2nd command searches in the same directory as above case but only for files that are greater than 10MB, but they should not have 'Metallica' as the starting of their filenames.
The 3rd command searches in the same directory for files that begin with 'Metallica' in their names or all the files that are greater than 10 MB in size.

-

The exec option is probably the most important feature of the find tool. The exec command allows you to execute a particular command on the results of the find command. A simple demonstration of this feature is shown below. Its upto your imagination to make maximum use of this feature. Suppose you wanted to see the details of the files (read, write, execute permission, file size, owner etc..) that have been returned as a search result you could do the following

$ find / - name 'Metallica*' -exec ls -l {\}\ \;

This command would find all the files on your system that begin with the letters 'Metallica' and would then execute the 'ls -l' command on these files. So basically you would be able to see the details of the files that were returned according to your search criteria.

The words following the -exec option is the command that you want to execute i.e. ls -l in this case.
{\}\ is basically an indicator that the filenames returned by the search should be substituted here.
\; is the terminating string, and is required at the end of the command


find test -type f  | xargs -i scp  '{}' root@192.168.0.1:'{}'
find ./test -type f  | xargs -i scp  '{}' root@192.168.0.1:~/'{}'
###60MB
find . -size +60000k -type f |xargs -i scp '{}' web1@dump.sample.net:~/
There is a difference between those two commands. The 1st just use the full path of the found file and copies it to the same path.
/etc/myapp/test.txt --> /etc/myapp/test.txt

The second will copy the full path to the home dir.
/etc/myapp/test.txt --> ~/etc/myapp/test.txt

Thanks for your help. Problem sovled



REFERENCES
http://www.codecoffee.com/tipsforlinux/articles/21.html
http://www.unix.com/shell-programming-scripting/113925-find-exec-scp-2.html

Thursday, March 31, 2011

File extension search and copy

SkyHi @ Thursday, March 31, 2011
find ./ -name "*.txt" -exec cp {} newfolder \;


But need to to find both .txt and .csv files
find ./ -name "*.[ct][sx][vt]" -exec cp {} newfolder \;
This command recursively finds from the current directory and subdirectories, files with names as per the below criteria.

After . the first character must be c or t (For .csv and .txt files)
After . the second character must be s or x (For .csv and .txt files)
After . the third character must be v or t (For .csv and .txt files)

Anyway, other combinations will also match (like .cxt) which I mentioned in my previous reply.

The exec in the above command copies all the found files (i.e, all .csv and .txt files we need) to the directory called newfolder.


ni@Januty:$ find . -iname "*.txt" -mtime -0.1 -exec cp {} ch5 \;


REFERENCES
http://www.unix.com/unix-advanced-expert-users/58363-file-extension-search-copy-3.html