Saturday, July 31, 2010

Python and vim: Make your own IDE

SkyHi @ Saturday, July 31, 2010

I prefer to use vim for most of my systems administration and programming related editing tasks. Aside from the usual argument that it will be present on any *nix system worth its silicon that you log in to, I choose it because of the succinct and expressive power of its syntax. While I am still learning new commands and techniques all the time, and while it is true that the learning curve to be anything resembling proficient is rather steep, few editors can boast such a wide range of actions in so few commands.

Right of out the box, however, vim isn’t as suited to editing Python code as it could be. In fact, it’s rather annoying to write Python code with an uncustomized instance of vim. What follows is a description of how to put into place what I see as the most essential features of the editor one chooses to write code, especially Python code, in as manifested with vim. With the following changes, you can create a highly customized and powerful IDE, allowing you to increase your productivity without purchasing a commercial offering.

Before the good stuff, a few requirements. You need to make sure you have vim-full and vim-python installed. Some systems come with vim-minimal, which is lacking in many advanced features. These packages should be in most repositories. Aside from these package installs, configuration changes described are made in ~/.vimrc. A quick word on the syntax of this file. Double quotation marks denote a line as a comment. I make it a rule to place a comment line before every configuration line or block thereof, so I have some sense later of what the change was for. Regarding Python files, I eventually also began to collect the related customizations into a file “~/.vim/ftplugin/python.vim”. This is loaded whenever Python type files are opened, and allows me to isolate related configurations.

  • Syntax highlighting
    Having code displayed with proper highlighting of the logical components makes reading that code easier to read and understand. Any decent code editor allows for coloring particular to a number of languages, and vim is no exception. To get this working, simply add:
    syntax on

    to your ~/.vimrc file. The actual colors will vary with your colorscheme. You can install new colorschemes (view hundreds here), as well as test out installed ones by typing


    The various colorschemes will appear as you tab.

  • Line numbering
    Having line numbers displayed with code makes it easier to reference and move around in. It’s easy to add. In ~/.vimrc:
    # Turn on line numbers:
    set number
    # Toggle line numbers and fold column for easy copying:
    nnoremap :set nonumber!:set foldcolumn=0

    This will allow you to toggle line numbers (as well as the spaces for the fold column described below) on and off by pressing . This makes it easy to copy out code without numbers.

  • Proper indentation and formatting
    In Python more than almost all languages, indentation is essential, both to get right and to preserve. While most programmers add indentation to a wide variety of languages for greater readability, in Python getting this wrong means a syntax error. The following directive will allow vim to properly keep indentation in your Python files, without you having to touch :
    filetype plugin indent on

    Additionally, you need to download the latest version of this script, placing it in ~/.vim/indent/python.vim. This will make indentation much less of a chore, and more of a pleasure. Be sure to also check out this blog post, wherein Henry PrĂȘcheur explains how to improve indentation for comments as well.

    To improve the formatting and display of improper indentation, as well as special highlighting for things like string formatting, download the latest version of this script, and place it in ~/.vim/syntax/python.vim. Then add this to your configuration:

    autocmd FileType python set complete+=k~/.vim/syntax/python.vim isk+=.,(
  • Shortcut complete syntax
    It can save time and your memory to be able to complete syntactic built-ins using a keyboard shortcut. [Edit: I had one method to get this working before, but it wasn't a good solution. For the right way to do this, check out this post.]
  • Access to documentation
    It is essential to be able to access built-in help quickly. This plugin allows you to view the built-in documentation for a given function by pressing “\pW” while over the word. This opens a new window, containing the related help documentation, if found.
  • Ability to run script being edited
    When debugging, I often need to make a small change, run the program, make another change, and so on. Having to write and close the file, then invoke it, is rather less than efficient. Simply add this to the ftplugin/python.vim file started above:
    " Execute file being edited with + e:
    map :w:!/usr/bin/env python %

    Now you can press + e to execute the Python file being edited. Once the code errors out or completes execution, output stays displayed until a key is pressed. Then you are returned to the code in the editor. Hard to imagine a shorter method.

  • Checker utilities and debugging
    pyflakes and pylint are two popular utilities for checking Python code for syntax errors, variable reference problems, and other signs of poor quality. If you haven’t tried them, install them both and see which you like best. Install the pyflakes and pylint packages for your system. For CentOS, you will need to add the EPEL repository to get access to these. Then add the functions listed here to your configuration. Now simply run “Pyflakes” and “Pylint” from within vim to view their output on your current file. Once you select the one you prefer, you could shorten access to it to a single keystroke, or even run it when you save your file.

    Once you have narrowed down the region you know a problem is present in, you will likely need to add some debugging statements and run the code repeatedly. Adding these functions into your .vimrc will allow you to add a PDB breakpoint with and remove all breakpoints with + .

  • Code folding
    As soon as your scripts start to encompass more than a few functions, much less multiple classes, it’s time to start checking out code folding. This feature allows the display of a class, method, or function as a two lines, one with the definition line and another showing length:

    With a simple key command, you can display or hide the entire block. There are a number of built-in commands to create and work with folds. I prefer using this plugin to allow vim to automatically know how to fold Python code. Once that is loaded, newly opened Python files will have all their classes and functions already folded, making it much faster to review files. + f toggles all folds, while f toggles the fold under the cursor.

  • Block navigation
    This script provides some advanced controls for selecting, navigating within, and acting upon blocks of code. You can jump to the top or bottom of a block, select it, even comment it out all with a few keystrokes.
  • Project navigation
    Nerdtree allows a window to be added to your vim display showing other files in your project. A “project” can either be the contents of your current directory, or you can define its members manually. This allows you to jump around your project, and open multiple files at once:

There are many, many more customizations possible concerning vim and Python. Check around the vim scripts pages if you have a particular itch to scratch. And if you know some good tricks that I didn’t list, please add them via comments!


Friday, July 30, 2010

Web Platform Benchmarks

SkyHi @ Friday, July 30, 2010
The following graph is in no way scientific. One day the engineers at just decided to author a "Hello, World" program on a variety of web development platforms to illustrate general relative performance. There are *many* more techniques for analyzing the performance of a web development platform then the one we used, and performance certainly isnt the only criteria when selecting a development platform either.

Obviously moves a tremendous amount of dynamic content daily, so we were interested in seeing how the different platforms stood up to what we were currently using. We posted our results here as a public service.

The benchmarks included mod_perl2, C Apache Modules, Static HTML, C, Perl, Python, Ruby, and Ruby On Rails CGI/FastCGI, and PHP.

All of the tests were run standalone on an unloaded webserver consisting of a 2.4 GHz Tyan Opteron 150,1 GB DDR 400 ECC RAM, running Linux, 64-bit.

In all cases, we followed the most straightforward and canonical install instructions we could find, compiled 64-bit from source with gcc -O2 (3.4.1), and always implemented the 'Hello, World' as dynamic text.

Webserver was Apache 2.2.

The test simply consisted of running ab -c 1000 http://localhost/etc.html on the same server. We would do this several times to come up with a good representational number.

We did also play with multiple requests but the overall *relative* performance was similar.

You may also be interested in our new GCC vs ICC Compiler Benchmarks - Perl Executable Performance


Thursday, July 29, 2010

Nerd tree Tips

SkyHi @ Thursday, July 29, 2010
Is there an easy way to add a file in nerd tree? Currently I go into my shell, add the file and then refresh the tree.


Activate the NERDTree and navigate to the directory in which the new file should live. Then press m to bring up the NERDTree Filesystem Menu and chose a for "add child node". Then simply enter the file's (or directory's name) and you're done.

How to copy (or duplicate) an entire folder in Vim using NERDTree?

NERDTree Menu. Use j/k/enter and the shortcuts indicated
> (a)dd a childnode
(m)ove the curent node
(d)elete the curent node
(c)copy the current node
m + c on the folder would duplicate it

How to jump back to NERDTree from file in tab?


Better Solution:

I've just started using the NERDTree vim plugin for my project.

I can't find the documentation for switching between opened tabs. Can anyone tell me the shortcut key[s] ?



An additional option (and my personal choice)beyond the ones listed by Michael Madsen:

gt = next tab

gT = previous tab

A quick check in :h tabs reveals it's CTRL-Page Down to cycle between tabs. You can also use the :tabnext command (:tabn for short).


I like to bind my vim navigation keys to switching between tabs. Here are the lines from my .vimrc file:

map   :tabn map   :tabp map   :tabnew 

That way, I can switch between tabs using the left and right buttons just like I normally would move the cursor, except I just hold the Control key as well.

  • Control+l moves to the next tab
  • Control+h moves to the previous tab
  • Control+n creates a new tab

Renaming or copying files and folder using NERDTree on Vim
NERDTree Menu. Use j/k/enter and the shortcuts indicated
> (a)dd a childnode
(m)ove the curent node
(d)elete the curent node
(c)copy the current node

Moving is the same as renaming.

vim and NERD Tree extension - adding a file

Activate the NERDTree and navigate to the directory in which the new file should live. Then press m to bring up the NERDTree Filesystem Menu and chose a for "add child node". Then simply enter the file's (or directory's name) and you're done.


from vim you can run shell commands. So in this case I use:

:!touch somefile.txt

and then hit 'r' to reload the nerdtree window.

The other thing to do is to just start the new file from within vim.

:e somefile.txt

One handy thing for this is that in my .vimrc I auto change the cwd to the directory my current file is in:

" Auto change the directory to the current file I'm working on

autocmd BufEnter * lcd %:p:h

This way if I'm editing a file and want another one in the same place the path is changed right there. Opening any file from NERDTree sets the directory to the one that file is in

Changing drive letter in NERDtree

I started using NERDtree plugin on a a windows platform. I can't seem to find a way to change the drive letter. Going up all the way on the directory tree doesn't lead to the drive letter. Any ideas? (Yes, I know, it's going to be a simple answer)

:NERDTree "drive letter":\


How do I insert text at beginning of a multi-line selection in VI/VIM?

For commenting blocks of code, I like the NERD Commenter plugin.

Select some text:

Shift-V the lines of text you want to comment....





Or just toggle the comment state of a line or block:


,c |NERDComToggleComment|
Toggles the comment state of the selected line(s). If the topmost selected line is commented, all selected lines are uncommented and vice versa.

,cc |NERDComComment|
Comments out the current line or text selected in visual mode.

,cn |NERDComNestedComment|
Same as |NERDComComment| but forces nesting.

,cm |NERDComMinimalComment|
Comments the given lines using only one set of multipart delimiters if

,cs |NERDComSexyComment|
Comments out the selected lines ``sexily''

,c$ |NERDComEOLComment|
Comments the current line from the cursor to the end of line.

,cA |NERDComAppendComment|
Adds comment delimiters to the end of line and goes into insert mode between them.

VIM: Close file without quiting VIM application?

This deletes the buffer (which translates to close the file)


vim indenting a block of code inward and outward

Ctrl + v

>> - increase indent of selected block

<< - decrease indent of selected block


Wednesday, July 28, 2010

Find Files By Access, Modification Date / Time Under Linux or UNIX

SkyHi @ Wednesday, July 28, 2010
I do not remember where I saved pdf and text files under Linux. I have downloaded files from the Internet a few months ago. How do I find my pdf or text files?

You need to use the find command. Each file has three time stamps, which record the last time that certain operations were performed on the file:

[a] access (read the file's contents) - atime
[b] change the status (modify the file or its attributes) - ctime
[c] modify (change the file's contents) - mtime

You can search for files whose time stamps are within a certain age range, or compare them to other time stamps.

You can use -mtime option. It returns list of file if the file was last accessed N*24 hours ago. For example to find file in last 2 months (60 days) you need to use -mtime +60 option.
  • -mtime +60 means you are looking for a file modified 60 days ago.
  • -mtime -60 means less than 60 days.
  • -mtime 60 If you skip + or - it means exactly 60 days.

So to find text files that were last modified 60 days ago, use
$ find /home/you -iname "*.txt" -mtime -60 -print

Display content of file on screen that were last modified 60 days ago, use
$ find /home/you -iname "*.txt" -mtime -60 -exec cat {} \; 

Count total number of files using wc command
$ find /home/you -iname "*.txt" -mtime -60 | wc -l

You can also use access time to find out pdf files. Following command will print the list of all pdf file that were accessed in last 60 days:
$ find /home/you -iname "*.pdf" -atime -60 -type -f

List all mp3s that were accessed exactly 10 days ago:
$ find /home/you -iname "*.mp3" -atime 10 -type -f

There is also an option called -daystart. It measure times from the beginning of today rather than from 24 hours ago. So, to list the all mp3s in your home directory that were accessed yesterday, type the command
$ find /home/you -iname "*.mp3" -daystart -type f -mtime 1 

  • -type f - Only search for files and not directories

-daystart option

The -daystart option is used to measure time from the beginning of the current day instead of 24 hours ago. Find out all perl (*.pl) file modified yesterday, enter:
find /nas/projects/mgmt/scripts/perl -mtime 1 -daystart -iname "*.pl"
You can also list perl files that were modified 8-10 days ago, enter:
To list all of the files in your home directory tree that were modified from two to four days ago, type:
find /nas/projects/mgmt/scripts/perl -mtime 8 -mtime -10 -daystart -iname "*.pl"

-newer option

To find files in the /nas/images directory tree that are newer than the file /tmp/foo file, enter:
find /etc -newer /tmp/foo
You can use the touch command to set date timestamp you would like to search for, and then use -newer option as follows
touch --date "2010-01-05" /tmp/foo
# Find files newer than 2010/Jan/05, in /data/images 
find /data/images -newer /tmp/foo
Read the man page of find command for more information:
man find

List ALL *.c File Accessed 30 Days Ago

Type the following command
find /home/you -iname "*.c" -atime -30 -type -f
Finds files older than 18months and newer than 24 months, cat’s the output to a CSV in the format:
/some/path/somewhere, size in bytes, Access Time, Modified Time

find /dir/dir -type f -mtime +540 -mtime -720 -printf \”%p\”,\”%s\”,\”%AD\”,|”%TD\”\\n > /dir/dir/output.csv



Linux: The differences between file times: atime (accessed time), ctime (changed time) and mtime (modified time).

Unless you have explicitly opted out with a noatime option mounting your Linux file system, there are generally 3 types of time on each and every file of Linux: atime or access time, ctime or change time, and mtime or modification time.

These are the differences between the 3 file system times:
  1. atime – access time, or the last accessed time of a file or directory, whenever you accessed, opened or touched it, the atime is updated to the current time. You can get the file atime value in PHP.
  2. ctime – change time, or the last changed time of the file or directory, whenever you change and update the file such as changing the file ownership or permissions or modifying the file content, the ctime of the file is updated to the current time. You can get the file ctime in PHP.
  3. mtime – modification time, or the last modified time of the file or directory, whenever you modify and update the content of the file. Modifying file ownerships or permissions doesn’t change mtime of the file. To get mtime of a file in PHP, you need a the php function filemtime.

Linux file ctime vs. mtime (differences):  
mtime is last time the content of the file is modified; while ctime is the last time the file is changed, not only regarding content, but also the meta information such as owner, group and permissions.


    Tuesday, July 27, 2010


    SkyHi @ Tuesday, July 27, 2010

    Let us start off simple:

    you have a large file ( txt, php, html, anything ) and you want to
    replace all the words "ugly" with "beautiful" because you just met your
    old friend Sue again and she/he is coming over for a visit.

    This is the command:

    sed -i 's/ugly/beautiful/g'

    Well, that command speaks for itself "sed" edits "-i in
    place ( on the spot ) and replaces the word "ugly with "beautiful"
    in the file "/home/bruno/old-friends/sue.txt"

    Now, here comes the real magic:

    you have a whole lot of files in a directory ( all about Sue ) and you
    want the same command to do all those files in one go because she/he is
    standing right at the door . .

    Remember the find command ? We will combine the two:

    find /home/bruno/old-friends -type f
    -exec sed -i 's/ugly/beautiful/g' {} \;

    in combination with the find command you can do all kind of nice
    tricks, even if you don't remember where the files are located !

    Aditionally I did find a little script on the net for if you often have to find and
    replace multiple files at once:


         for fl in *.php; do

         mv $fl $fl.old

         sed 's/FINDSTRING/REPLACESTRING/g' $fl.old > $fl

         rm -f $fl.old


    just replace the "*.php", "FINDSTRING" and "REPLACESTRING"
    make it executable and you are set.

    changed a www address in 183 .html files in one go with this little
    script . . . but note that you have to use "escape-signs" ( \ ) if
    there are slashes in the text you want to replace, so as an example:
    's/\/images/\/linux/g' to change to

    For the lovers of perl I also found this one:

    perl -e "s/old_string/new_string/g;" $(find DirectoryName -type f)

    it leaves "traces", e.g it backs up the old file with a .save extension
    . . . so is not really effective when Sue comes around ;-/