Tuesday, June 1, 2010

time stamp of file create

— SkyHi @ Tuesday, June 01, 2010
perl version for inodetime



Code:


<!--ntstart-->#!/usr/contrib/bin/<acronym title="Practical Extraction and Report Language">perl</acronym>
#^###################################################################
#^ $Header: inodetime.pl,v 1.1 97/01/20 10:17:00 http srvr? $
#^###################################################################
#^ PROGRAM DESCRIPTION
#^ -------------------
#^ This program prints the modification times and names of files.
#^ It uses the following format: inodetime.pl filename
#^ It will accept: inodetime.pl filename1 filename2 filename3
#^ inodetime.pl /tmp/file*
#^ The format of the output is: YYYYMMDDhhmmss filename
#^ example:
#^ $ inodetime.pl /tmp/t*
#^ 19961115105425 /tmp/test.sql
#^ 19970116113616 /tmp/tststat.pl
#^
#^ TRICK To Remember The Program Name
#^ ----------------------------------
#^ inodetime == I No De Time
#^ ^^ ^ ^
#^ KSG
#^###################################################################

############################################
# Get the (next) input from the command line
############################################
while ($curfile = $ARGV[0])
{
#################################################
# Do following code block only if $curfile exists
#################################################
if (-e $curfile)
{
###############################
# stat structure into variables
###############################
($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
$atime,$mtime,$ctime,$blksize,$blocks)
= stat("$curfile");
###############################
# time structure into variables
###############################
local($sec,$min,$hr,$day,$mon,$yr,$wday,@dntcare) = localtime($mtime);
$yr = ($yr>=70) ? $yr+1900 : $yr+2000;
$yr="$yr";
$mon = (++$mon < 10) ? "0$mon" : "$mon";
$day = ($day < 10) ? "0$day" : "$day";
$hr = ($hr < 10) ? "0$hr" : "$hr";
$min = ($min < 10) ? "0$min" : "$min";
$sec = ($sec < 10) ? "0$sec" : "$sec";
##################################################################
# Rearrange in the YYYYMMDDhhmmss format and assign to $dte variable
##################################################################
$dte = join('',$yr,$mon,$day,$hr,$min,$sec);
######################################
# Print modification date and filename
######################################
print ("$dte $curfile\n");
}
########################################
# Shift to next position in command line
########################################
shift (@ARGV);
}


#~###################################################################
#~###################################################################
#~
#~ $Log: inodetime.pl,v $
#~ Revision 1.1 97/01/20 10:17:00 10:17:00 http ()
#~ reversed output
#~ from: filename date
#~ to: date filename
#~
#~ Revision 1.0 97/01/17 12:37:32 12:37:32 http ()
#~ Initial revision
#~
#~###################################################################
#~###################################################################



REFERENCES
http://www.unix.com/unix-dummies-questions-answers/14206-time-stamp-file-create.html

Monday, May 31, 2010

Installing TrueCrypt on Ubuntu

— SkyHi @ Monday, May 31, 2010

This guide will go through installing TrueCrypt encryption software from a terminal in order to install the command line version (in cases were desktop access is unavailable) and as a GUI for users that require a simple working interface.


Installing Command Line Version (from terminal)

Go to download page and download the “Mac OS X/Linux” version


http://www.truecrypt.org/downloads2


The required software for building TrueCrypt are listed below:


- GNU Make – GNU C++ Compiler 4.0 or compatible


- pkg-config


- FUSE library and header files (available at http://fuse.sourceforge.net and http://code.google.com/p/macfuse)


- RSA Security Inc. PKCS #11 Cryptographic Token Interface (Cryptoki) 2.20 header files (available at ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20) located in a standard include path or in a directory defined by the environment variable ‘PKCS11_INC’.


Install them using the following:


apt-get install g++ pkg-config libfuse-dev libfuse2 fuse-utils


Make sure the fuse module is added to your kernel and that it is loaded every time your system starts. You can do that by editing the /etc/modules file and appending “fuse” (without quotes) to it. Finally, extract the tar.gz archive.


sudo modprobe fuse

sudo nano /etc/modules

OR

su echo fuse >> /etc/modules

tar xvf TrueCrypt\ 6.1a\ Source.tar.gz

cd truecrypt-6.1a-source/

make NOGUI=1 WXSTATIC=1


In case of a compile error which refers to missing file “pkcs11.h” you can download it manually using:


wget ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20/pkcs11.h

wget ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20/pkcs11f.h

wget ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v2-20/pkcs11t.h


Finally you can copy the executable to /usr/local/bin by:


cp Main/truecrypt /usr/local/bin/


Installing GUI Version

Installing the GUI version is extremely simple. Get/extract/execute the file using:


wget http://www.truecrypt.org/download/truecrypt-6.1a-ubuntu-x86.tar.gz

tar xvf truecrypt-6.1a-ubuntu-x86.tar.gz

./truecrypt-6.1a-setup-ubuntu-x86

or

wget http://www.truecrypt.org/download/truecrypt-6.1a-ubuntu-x64.tar.gz

tar xvf truecrypt-6.1a-ubuntu-x64.tar.gz

./truecrypt-6.1a-setup-ubuntu-x64




REFERENCES
http://www.itsolutionskb.com/2009/04/installing-truecrypt-on-ubuntu/

TrueCypt Create Hidden Volume

— SkyHi @ Monday, May 31, 2010

It may happen that you are forced by somebody to reveal the password to an encrypted volume. There are many situations where you cannot refuse to reveal the password (for example, due to extortion). Using a so-called hidden volume allows you to solve such situations without revealing the password to your volume.


The layout of a standard TrueCrypt volume before and after a hidden volume was created within it.


The layout of a standard TrueCrypt volume before and after a hidden volume was created within it.




The principle is that a TrueCrypt volume is created within another TrueCrypt volume (within the free space on the volume). Even when the outer volume is mounted, it is impossible to prove whether there is a hidden volume within it or not*, because free space on any TrueCrypt volume is always filled with random data when the volume is created** and no part of the (dismounted) hidden volume can be distinguished from random data. Note that TrueCrypt does not modify the file system (information about free space, etc.) within the outer volume in any way.




The password for the hidden volume must be substantially different from the password for the outer volume. To the outer volume, (before creating the hidden volume within it) you should copy some sensitive-looking files that you actually do NOT want to hide. These files will be there for anyone who would force you to hand over the password. You will reveal only the password for the outer volume, not for the hidden one. Files that really are sensitive will be stored on the hidden volume.


A hidden volume can be mounted the same way as a standard TrueCrypt volume: Click Select File or Select Device to select the outer/host volume (important: make sure the volume is not mounted). Then click Mount, and enter the password for the hidden volume. Whether the hidden or the outer volume will be mounted is determined by the entered password (i.e., when you enter the password for the outer volume, then the outer volume will be mounted; when you enter the password for the hidden volume, the hidden volume will be mounted).


TrueCrypt first attempts to decrypt the standard volume header using the entered password. If it fails, it loads the area of the volume where a hidden volume header can be stored (i.e. the bytes 65536–131071, which contain solely random data when there is no hidden volume within the volume) to RAM and attempts to decrypt it using the entered password. Note that hidden volume headers cannot be identified, as they appear to consist entirely of random data. If the header is successfully decrypted (for information on how TrueCrypt determines that it was successfully decrypted, see the section Encryption Scheme), the information about the size of the hidden volume is retrieved from the decrypted header (which is still stored in RAM), and the hidden volume is mounted (its size also determines its offset).


A hidden volume can be created within any type of TrueCrypt volume, i.e., within a file-hosted volume or partition/device-hosted volume (requires administrator privileges). To create a hidden TrueCrypt volume, click on Create Volume in the main program window and select Create a hidden TrueCrypt volume. The Wizard will provide help and all information necessary to successfully create a hidden TrueCrypt volume.


When creating a hidden volume, it may be very difficult or even impossible for an inexperienced user to set the size of the hidden volume such that the hidden volume does not overwrite data on the outer volume. Therefore, the Volume Creation Wizard automatically scans the cluster bitmap of the outer volume (before the hidden volume is created within it) and determines the maximum possible size of the hidden volume.***


If there are any problems when creating a hidden volume, refer to the chapter Troubleshooting for possible solutions.





Note that it is also possible to create and boot an operating system residing in a hidden volume (see the section Hidden Operating System).








* Provided that all the instructions in the TrueCrypt Volume Creation Wizard have been followed and provided that the requirements and precautions listed in the subsection Security Requirements and Precautions Pertaining to Hidden Volumes are followed.

** Provided that the options Quick Format and Dynamic are disabled and provided that the volume does not contain a filesystem that has been encrypted in place (TrueCrypt does not allow the user to create a hidden volume within such a volume). For information on the method used to fill free volume space with random data, see chapter Technical Details, section TrueCrypt Volume Format Specification.

*** The wizard scans the cluster bitmap to determine the size of the uninterrupted area of free space (if there is any) whose end is aligned with the end of the outer volume. This area accommodates the hidden volume and therefore the size of this area limits the maximum possible size of the hidden volume. On Linux and Mac OS X, the wizard actually does not scan the cluster bitmap, but the driver detects any data written to the outer volume and uses their position as previously described.

REFERENCES
http://www.truecrypt.org/docs/?s=hidden-volume

How to Encrypt Your USB Flash Drive Using TrueCrypt

— SkyHi @ Monday, May 31, 2010

USB flash drives are becoming cheaper and cheaper everyday. Some companies are even giving them away. When they first arrived on the scene, most of the drives had a capacity of under 1GB. But now, you can find 2GB to 4GB drives almost everywhere, including your local drugstore chain. At these sizes, they can actually be useful. You can use it to store your music, pictures, videos, or documents. Some even use it to store bootable operating systems like Linux. I use it to store a text file that contains the passwords for all of my online accounts, such as for my online bank accounts, my Amazon account, credit card accounts, etc. And since the flash drives are so portable, it makes sense to have one. However, since they ARE so portable, they can easily be lost, stolen, or misplaced. If you are like me, and store personal information on your flash drive, information that you don't want to fall into the wrong hands, then you need to encrypt your flash drive. By encrypting your flash drive, the files contained within it become password protected and can only be accessed by you or someone who knows your password.



There are many different applications that help you encrypt your flash drive. Some drive manufacturers include encryption applications on the flash drive. In this tutorial, I will show you how to encrypt your portable USB flash drive using my favorite freeware application, TrueCrypt.












Difficulty: Moderately Easy

Instructions





Things You'll Need:



  • USB Flash drive

  • Latest stable version of TrueCrypt




  1. Step 1


    Using TrueCrypt, you create a password protected encrypted file that is stored on the flash drive. This encrypted file acts as a "container", within which all the files you want encrypted are stored. When you connect your flash drive into a PC, this "container" gets mounted as a separate hard drive (provided you enter the correct password). And now, everything you save into this separate hard drive is encrypted automatically. This is where TrueCrypt really shines, providing transparent, real-time encryption. Plus, you don't need TrueCrypt to be installed on the local computer.



  2. Step 2


    Download the latest stable version of TrueCrypt here: http://www.truecrypt.org/downloads.php



  3. Step 3







     







    Install the software on your local computer (accepting all the default options)



  4. Step 4


    Connect your USB flash drive to your computer. For this tutorial, let's assume that it is assigned drive letter G:\



  5. Step 5







     







    Start the TrueCrypt application. Click on the Create Volume button to start the TrueCrypt Volume Creation Wizard. This is where you create the "container".



  6. Step 6







     







    Select Create a file container (default option) and click on Next. This brings you to the Volume Type window. Here you can specify if you want your "container" to be a standard, visible file or if you want to create a hidden "container" (essentially a "container" within a "container"). For this tutorial, we'll select the default option, Standard TrueCrypt Volume, and click on Next.



  7. Step 7







     







    This brings you to the Volume Location window. Here you specify the filename and location of the "container". For this tutorial, let's call the container "MyCrypt". And since your flash drive is mounted as the G:\ drive, specify your location and filename as G:\MyCrypt, placing the container in the root of the flash drive. Click Next.



  8. Step 8







     







    Next you need to select the Encryption Algorithm and Hash Algorithm. I won't go into the details of the differences between the different options, their pros and cons. That would turn this tutorial into a book. For this tutorial, we'll leave the defaults, as they should be sufficient. Click Next.



  9. Step 9







     







    Next, you need to choose the size of the "container". This depends on the size of your flash drive and how much info you want to encrypt. Personally, I would suggest leaving anywhere between 10% to 20% of the drive unencrypted so that you have room for the TrueCrypt application files (about 6MB) as well as unimportant files that you might want to share or just don't need encrypted. For this tutorial, using a 1GB flash drive, we'll set the "container" to be 850MB. Click Next.



  10. Step 10







     







    Next, specify the password you want to use to access and mount this "container". Select a strong password, that would be easy for you to remember and hard for anyone else to figure out. A strong password usually consists of at least 20 characters, and uses a combination of letters (both lower and upper case), and numbers. But at a minimum, it should consist of 8 characters. Click Next after you enter your password.



  11. Step 11







     







    Next, you are ready to "format" the container. You can select the type of Filesystem and Cluster. For this tutorial, leave the default values. Move your mouse randomly within the Volume Format window to generate the encryption keys. Don't worry; you are not going to have to remember these keys. When ready, click on Format to start. Depending on the size of the "container" (chosen in step 8), this may take up to 5 minutes.



  12. Step 12







     







    Once the format successfully completes, you will get a pop up indicating that the "container" has been created. Click OK.



  13. Step 13







     







    Close out of the TrueCrypt application. Using Windows Explorer or My Computer, navigate to the TrueCrypt directory, usually under C:\Program Files. Copy the entire C:\Program Files\TrueCrypt directory to the root of your flash drive. At this point you should have the MyCrypt "container" and the TrueCrypt directory (with application files) on your flash drive.



  14. Step 14


    Finally, using notepad, create a file called autorun.inf and enter the following:

    [autorun]
    action=Mount TrueCrypt volume
    open=TrueCrypt\TrueCrypt.exe /q background /e /m rm /v "MyCrypt"
    shell\start=Start TrueCrypt
    shell\start\command=TrueCrypt\TrueCrypt.exe
    shell\dismount=Dismount all TrueCrypt volumes
    shell\dismount\command=TrueCrypt\TrueCrypt.exe /q /d

    The autorun.inf file, with the above entries, allows you to be prompted to mount the encrypted "container", every time you connect your flash drive.



  15. Step 15







     







    Save this file into the root directory of your flash drive.



  16. Step 16







     







    That's it! Now, every time you connect your flash drive, you will be asked if you want to mount your encrypted "container". Select Mount TrueCrypt volume and click OK.



  17. Step 17







     







    Next, you will be prompted to enter in the password you created for your encrypted "container". Enter your password and click OK.



  18. Step 18







     







    Your encrypted "container" will be mounted as a drive using the next available drive letter. In this case, it is the H:\ drive



  19. Step 19







     







    Now, every time you put a file into the H:\ drive, it will be encrypted automatically. To "disconnect" the drive, right-click on the TrueCrypt icon in your taskbar and select Dismount.

















Tips & Warnings

  • Hopefully, I made the steps to create an encrypted drive easy. Having an encrypted drive will give you the assurance that if you lost your flash drive, the personal information stored in the encrypted drive will never be exposed.
  • TrueCrypt and the TrueCrypt logo are registered trademarks of the TrueCrypt Foundation
  • These steps and other tips can also be found on my blog http://nyctechtips.com
REFERENCES
http://www.ehow.com/how_2298754_usb-flash-drive-using-truecrypt.html

Encrypt Your USB Stick With Truecrypt 6.0

— SkyHi @ Monday, May 31, 2010
Last year I wrote an article on the benefits of encrypting your PC folders with Truecrypt and I also briefly touched on being able to encrypt your USB stick with Truecrypt. Well, the other day I received a nice new 2GB USB stick as a freebie and so I decided to install John Haller’s Portable Apps on it. But first I headed on over to the Truecrypt website to install the newly updated 6.0 encryption program.

Encryption is absolutely essential, especially if you’re the kind of person that carries their USB stick around as if it’s your car keys or your lipstick. USB sticks are so small that they are easily lost and they are also easily stolen. Just think of all the information that gets stored on one of these things. The British Ministry of Defence has lost 131 of them since 2004! A friend of mine constantly drops his in the street when he walks his dog and his dog keeps walking back to pick it up! So it definately pays to take the time and have encryption.


Also, look at it this way. If someone found your USB stick and it was unencrypted, they would have access to your Firefox browser (with access to your private bookmarks, including online banking), your private files, your portable FTP program (with the settings to your website), passwords, emails, IM contacts and much more. If the person was honest, it might not be so bad, but if the person wasn’t honest….well then it could be catastrophic for you. Identity theft would only be the start of your problems.


The only problem with using Truecrypt for your encryption though is that you need to have administrator privileges on the computer in which you’re plugging your encrypted USB stick into. So this would be no good for internet cafes for example. This would only be good if you were travelling between multiple trusted personal and work computers and you wanted insurance against theft or loss while travelling around.


OK, let’s get insured.


Step One – Download Truecrypt 6.0


The obvious first step. Head on over and download the encryption program. BUT since you are putting this on your USB stick, you do NOT install this anywhere. When you double-click the “exe” file, you will see this. Choose the second option. This will just unpack the files onto your computer and not install anything.



When the files are unpacked, move the whole lot over to your USB stick. Once they are sitting on your USB stick, again don’t install anything. Just leave them there.


Step Two – Turn Truecrypt Into “Traveller Mode”


In your USB stick folder, double-click the “Truecrypt.exe” file and this opens up the main screen. Go to “tools” then “traveler disk setup”. That brings up this :


Traveller Mode


Change “create traveler disk files at” to whichever PC drive you have your USB stick in at the moment. Have the box underneath ticked. As to the Autorun configuration, well that’s down to personal preference so you decide. When you’ve done everything, press create. You’ll get a message a few moments later telling you that the traveler mode has been successfully created.


Step Three – Move Your Desired Programs Into Your USB stick


Now move your desired programs and files into your USB stick. As I said, I decided to put John Haller’s excellent PortableApps onto my USB stick. So I downloaded and moved all of them onto the USB stick next to the Truecrypt program as well as some documents. Notice we haven’t encrypted anything yet. That comes next.


Step Four – Create An Encrypted Area


OK, now we’re going to do the encryption. Go to the Truecrypt folder in your USB stick and double-click “Truecrypt Format.exe”. That brings up this screen.



Choose “create a file container” and follow these instructions from step 3 to make an encrypted container. I’m not going to go through it all here as it’s a long process. One important point though – do NOT fill up the whole USB stick. I would only make a container that fills up at most 75% of your USB stick. Leave the rest free and unencrypted. It’s always good to have some free space.


Before anybody points out option 2 which encrypts the entire USB stick, everyone I have talked to about this recommends that this shouldn’t be done because undoing it later is tricky and it can potentially mess up your USB drive. Doing it my way is better because a) you have some free space left for unencrypted files and b) if you don’t want Truecrypt anymore, you can just delete the whole lot. Much easier.


Step Five – Move The Programs Into The Encrypted Area



Once the encrypted container has been made, you will then have three things sitting in your USB stick – the folder with your programs / files, the folder with the Truecrypt application and the encrypted container you have just made. Now what you are going to do is move your programs and files INSIDE the encrypted container.


You do this by double-clicking “Truecrypt.exe” and “mounting” (choosing) on another drive, the encrypted container you have just made. This opens it up and unencrypts it. When it opens up, just drag your files and programs inside with your mouse. When you’re done, just dismount the container which closes it back up again and heavily encrypts it. No-one can get inside now without knowing the password. So I hope you chose a very good password!


Don’t forget to delete the programs and files which are sitting outside the encrypted container – the ones you just dragged inside the container are copies. If it reassures you, mount the container again, open it up and peek inside. You’ll see your files and apps inside all safe and sound.


Step Six – In Summary


So let’s recap what we’ve done here. We haven’t encrypted the whole USB stick. All we’ve done is put an encrypted container on the USB stick along with the Truecrypt encryption program to encrypt and unencrypt your files. If anyone was to steal the USB stick or find it (if you dropped it), all they would find on the drive would be an encrypted container which, without the password, is absolutely useless and unbreakable.


You also might want to put (on the unencrypted part of the drive) a text file with your name, email address and phone number so if a Good Samaritan were to come along, they would have a chance to do the right thing and return your property to you. A little incentive could also be mentioned depending on how desperate you are to get your property back.


Do you like to use any other encryption programs other than Truecrypt? What particular USB stick protection and detection methods have worked for you? Let’s hear about them in the comments!


REFERENCES

http://www.makeuseof.com/tag/encrypt-your-usb-stick-with-truecrypt-60/

Creating a Basic TrueCrypt Volume on a USB Drive

— SkyHi @ Monday, May 31, 2010
We’ve talked about TrueCrypt a couple of times in the past (see Setup and Configuration of My USB Drive and Installing TrueCrypt). I thought it would be worth creating a walk-through showing you how to create your own basic TrueCrypt encrypted volume on a USB drive.

truecryptvolume000.pngTo start the process, make sure that TrueCrypt is installed on your computer system. If not, please see Installing TrueCrypt for a tutorial on how to install TrueCrypt. Double click on the TrueCrypt icon to start the TrueCrypt program.


truecryptvolume001.pngOnce TrueCrypt is running, you will see a window similar to the one on the left. Click on the Create Volume button in the middle of the window on the left hand side. This starts the TrueCrypt Volume Creation Wizard.


truecryptvolume002.pngIn the wizard window, select Create a standard TrueCrypt volume and then click on the Next > button.


The Volume Location window appears next. This lets you indicate where you want this volume to reside. Since we are going to be encrypting an entire USB drive, you want to click on the Select Device… button.


truecryptvolume003.pngYou can now select the device that you want to encrypt. This may be the most confusing part for most people since they are not typically acccustomed to how the drives are referenced. If you look under the Drive: column, you will see the drive letter that you are probably used to seeing. Select the letter of the drive you want to encrypt. Then, select the entry directly above the drive you want to encrypt. This will encrypt the entire drive and not just the partition. In the example that I am using, I want to encrypt the USB drive which is presently on the F: drive. So, I will chose Harddisk 2: to encrypt. Then click on the OK button.


Warning: Be very careful about the drive you select. If you select the wrong one, you could perminantly lose your data!


lost001.pngYou should now be back at the Volume Location window and your location should appear. Click the Next > button.


truecryptvolume004.pngYou are now prompted for encryption options. Since we are just looking at creating a very basic volume, we will select AES for the Encryption Algorithm and RIPEMD-160 for the Hash Algorithm (I’ll get into what these are all about in a later article, I promise). Click on the Next > button.


truecryptvolume007.pngThe Volume Size window now appears. Since we are encrypting an entire USB drive, there are no options to select at this point. Click Next > to continue.


truecryptvolume008.pngYou are now asked for a password in order to access the volume. Be certain to select a good password (This sounds like it would make a good article as well). Enter it in both the Password: and the Confirm: boxes and then click Next >.


Warning: Do not forget your password because there is no known way of retreiving the password if it is lost. This means that you will never be able to get the data back.


truecryptvolume011.pngIn the Volume Format window, leave all of the default settings and click the Format button.


truecryptvolume012.pngYou are not presented with a warning window. Click on the OK button to proceed.


truecryptvolume013.pngTrueCrypt will now start to format the volume. The length of time that the formatting process takes will vary depending on how big the USB drive is and how powerful your computer is. Please be patient while this finishes.


truecryptvolume014.pngWhen the formatting is done, you will now have a TrueCrypt encrypted USB drive. Click on the Exit button to finish the wizard and then you can click on the Exit button to close TrueCrypt.


In upcoming articles, I will be talking about how to mount and dismount a TrueCrypt volume along with how to pick some more advanced options with the TrueCrypt system.


REFERENCES

http://dailycupoftech.com/?page_id=100


The top 10 tricks of Perl one-liners

— SkyHi @ Monday, May 31, 2010



I’m a recovering perl hacker. Perl used to be far and away my language of choice, but these days I’m more likely to write new code in Python, largely because far more of my friends and coworkers are comfortable with it.



I’ll never give up perl for quick one-liners on the command-line or in one-off scripts for munging text, though. Anything that lasts long enough to make it into git somewhere usually gets rewritten in Python, but nothing beats perl for interactive messing with text.



Perl, never afraid of obscure shorthands, has accrued an impressive number of features that help with this use case. I’d like to share some of my favorites that you might not have heard of.


One-liners primer


We’ll start with a brief refresher on the basics of perl one-liners before we begin. The core of any perl one-liner is the -e switch, which lets you pass a snippet of code on the command-line:

perl -e 'print "hi\n"' prints “hi” to the console.


The second standard trick to perl one-liners are the -n and -p flags. Both of these make perl put an implicit loop around your program, running it once for each line of input, with the line in the $_ variable. -p also adds an implicit print at the end of each iteration.


Both of these use perl’s special “ARGV” magic file handle internally. What this means is that if there are any files listed on the command-line after your -e, perl will loop over the contents of the files, one at a time. If there aren’t any, it will fall back to looping over standard input.



perl -ne 'print if /foo/' acts a lot like grep foo, and perl -pe 's/foo/bar/' replaces foo with bar



Most of the rest of these tricks assume you’re using either -n or -p, so I won’t mention it every time.


The top 10 one-liner tricks


Trick #1: -l


Smart newline processing. Normally, perl hands you entire lines, including a trailing newline. With -l, it will strip the trailing newline off of any lines read, and automatically add a newline to anything you print (including via -p).



Suppose I wanted to strip trailing whitespace from a file. I might naïvely try something like


  perl -pe 's/\s*$//'

The problem, however, is that the line ends with "\n", which is whitespace, and so that snippet will also remove all newlines from my file! -l solves the problem, by pulling off the newline before handing my script the line, and then tacking a new one on afterwards:


  perl -lpe 's/\s*$//'<br />


Trick #2: -0

Occasionally, it’s useful to run a script over an entire file, or over larger chunks at once. -0 makes -n and -p feed you chunks split on NULL bytes instead of newlines. This is often useful for, e.g. processing the output of find -print0. Furthermore, perl -0777 makes perl not do any splitting, and pass entire files to your script in $_.
  find . -name '*~' -print0 | perl -0ne unlink<br />

Could be used to delete all ~-files in a directory tree, without having to remember how xargs works.


Trick #3: -i

-i tells perl to operate on files in-place. If you use -n or -p with -i, and you pass perl filenames on the command-line, perl will run your script on those files, and then replace their contents with the output. -i optionally accepts an backup suffix as argument; Perl will write backup copies of edited files to names with that suffix added.
  perl -i.bak -ne 'print unless /^#/' script.sh<br />

Would strip all whole-line commands from script.sh, but leave a copy of the original in script.sh.bak.


Trick #4: The .. operator

Perl’s .. operator is a stateful operator — it remembers state between evaluations. As long as its left operand is false, it returns false; Once the left hand returns true, it starts evaluating the right-hand operand until that becomes true, at which point, on the next iteration it resets to false and starts testing the other operand again.


What does that mean in practice? It’s a range operator: It can be easily used to act on a range of lines in a file. For instance, I can extract all GPG public keys from a file using:


     perl -ne 'print if /-----BEGIN PGP PUBLIC KEY BLOCK-----/../-----END PGP PUBLIC KEY BLOCK-----/' FILE<br />

Trick #5: -a

-a turns on autosplit mode – perl will automatically split input lines on whitespace into the @F array. If you ever run into any advice that accidentally escaped from 1980 telling you to use awk because it automatically splits lines into fields, this is how you use perl to do the same thing without learning another, even worse, language.


As an example, you could print a list of files along with their link counts using


ls -l | perl -lane 'print "$F[7] $F[1]"'<br />


Trick #6: -F

-F is used in conjunction with -a, to choose the delimiter on which to split lines. To print every user in /etc/passwd (which is colon-separated with the user in the first column), we could do:
perl -F: -lane 'print $F[0]' /etc/passwd<br />

Trick #7: \K

\K is undoubtedly my favorite little-known-feature of Perl regular expressions. If \K appears in a regex, it causes the regex matcher to drop everything before that point from the internal record of “Which string did this regex match?”. This is most useful in conjunction with s///, where it gives you a simple way to match a long expression, but only replace a suffix of it.

Suppose I want to replace the From: field in an email. We could write something like


perl -lape 's/(^From:).*/$1 Nelson Elhage <nelhage\@ksplice.com>/'<br />


But having to parenthesize the right bit and include the $1 is annoying and error-prone. We can simplify the regex by using \K to tell perl we won’t want to replace the start of the match:


perl -lape 's/^From:\K.*/ Nelson Elhage <nelhage\@ksplice.com>/'<br />

Trick #8: $ENV{}

When you’re writing a one-liner using -e in the shell, you generally want to quote it with ‘, so that dollar signs inside the one-liner aren’t expanded by the shell. But that makes it annoying to use a ‘ inside your one-liner, since you can’t escape a single quote inside of single quotes, in the shell.


Let’s suppose we wanted to print the username of anyone in /etc/passwd whose name included an apostrophe. One option would be to use a standard shell-quoting trick to include the ‘:


perl -F: -lane 'print $F[0] if $F[4] =~ /'"'"'/' /etc/passwd<br />


But counting apostrophes and backslashes gets old fast. A better option, in my opinion, is to use the environment to pass the regex into perl, which lets you dodge a layer of parsing entirely:


env re="'" perl -F: -lane 'print $F[0] if $F[4] =~ /$ENV{re}/' /etc/passwd<br />


We use the env command to place the regex in a variable called re, which we can then refer to from the perl script through the %ENV hash. This way is slightly longer, but I find the savings in counting backslashes or quotes to be worth it, especially if you need to end up embedding strings with more than a single metacharacter.


Trick #9: BEGIN and END

BEGIN { ... } and END { ... } let you put code that gets run entirely before or after the loop over the lines.


For example, I could sum the values in the second column of a CSV file using:


perl -F, -lane '$t += $F[1]; END { print $t }'<br />

Trick #10: -MRegexp::Common

Using -M on the command line tells perl to load the given module before running your code. There are thousands of modules available on CPAN, numerous of them potentially useful in one-liners, but one of my favorite for one-liner use is Regexp::Common, which, as its name suggests, contains regular expressions to match numerous commonly-used pieces of data.


The full set of regexes available in Regexp::Common is available in its documentation, but here’s an example of where I might use it:



Neither the ifconfig nor the ip tool that is supposed to replace it provide, as far as I know, an easy way of extracting information for use by scripts. The ifdata program provides such an interface, but isn’t installed everywhere. Using perl and Regexp::Common, however, we can do a pretty decent job of extracing an IP from ip’s output:


ip address list eth0 | \<br />  perl -MRegexp::Common -lne 'print $1 if /($RE{net}{IPv4})/'<br />





So, those are my favorite tricks, but I always love learning more. What tricks have you found or invented for messing with perl on the command-line? What’s the most egregious perl “one-liner” you’ve wielded, continuing to tack on statements well after the point where you should have dropped your code into a real script?





perl -lne s/rebooting//i


Tired of babysitting your systems while you reboot to install updates? Eliminate reboots with Ksplice Uptrack, and spend that time honing your perl scripts instead!



REFERENCES

http://blog.ksplice.com/2010/05/top-10-perl-one-liner-tricks/