w3m - A Text Based Commandline Web Browser

w3m is a World Wide Web text based client. It will display hypertext markup language (HTML) documents containing links to files residing on the local system, as well as files residing on remote systems. It can display HTML tables and frames. In addition, it can be used as a "pager" in much the same manner as "more" or "less". Current versions of w3m run on Unix (Solaris, SunOS, HP-UX, Linux, FreeBSD, and EWS4800) and on Microsoft Windows 9x/NT.

Linux users can install the package from their respective repositories. Following is the example of installation in ubuntu and debian based linux.

samar@Techgaun:~$ sudo apt-get install w3m

Using the w3m browser is pretty straightforward. At start up, w3m will load any local file or remote URL specified at the command line. An example usage is as below:

samar@Techgaun:~$ w3m http://www.techgaun.com

You can see the whole list of available operation by pressing H(Shift + h) and you will know how comprehensive this seemingly simple command line browser actually is. w3m supports all kind of features we expect from a web browser such as hyperlink navigations, tabbed browsings, file I/O operations, bookmarking, and searching.

w3m can also be used as a pager and for translating HTML files. Taken directly from w3m manpage, following are the examples:

To use w3m as a pager:

samar@Techgaun:~$ ls | w3m

To use w3m to translate HTML files:

samar@Techgaun:~$ cat foo.html | w3m -T text/html

or

samar@Techgaun:~$ cat foo.html | w3m -dump -T text/html >foo.txt

Read more...

How To Disable Password Prompts For sudo In Ubuntu

If you are one of those linux users who very frequently use the sudo command, you might have been annoyed of entering passwords each time you use this command. However with a very simple tweak, you can change this behaviour and disable the password prompts for the sudo command.

A bit of warning though, do not modify the default behavior of asking for passwords since it would drastically compromise security of your system. Following is the warning given by ubuntu help.

If you disable the sudo password for your account, you will seriously compromise the security of your computer. Anyone sitting at your unattended, logged in account will have complete Root access, and remote exploits become much easier for malicious crackers.

Now you are aware of warning, lets see how this can be done.We need to edit the /etc/sudoers file. Lets first open the file in safe editable mode using the following command:

samar@Techgaun:~$ sudo visudo

Using visudo for editing the /etc/sudoers lets us locate the possible errors that may occur while editing the file so always use visudo.

Following is the format of disabling password prompts for specific user.

<username> ALL=NOPASSWD: ALL

So if we are to disable password prompts for the user samar, you can make a new line with following entry:

samar ALL=(ALL)NOPASSWD: ALL

In case you want to let all the users with admin privilege use the sudo command without having to give the password, you can edit the line that says %admin ALL=(ALL) ALL to the following:

%admin ALL=(ALL)NOPASSWD: ALL

Once you add such line for appropriate user, press Ctrl + x and save the changes. You will either have to log out and login back or restart the shell to see the modification in effect.

I hope this is useful. :)


Read more...

nmbscan - Network Shares Scanner Based On NMB/SMB/NetBIOS Protocol

NMB Scanner scans the shares of a NetBIOS/SMB network, using the NMB/SMB/NetBIOS protocols. It is useful for acquiring information on a local area network for such purposes as security auditing.

It can obtain such information as NMB/SMB/NetBIOS/Windows hostname, IP address, IP hostname, ethernet MAC address, Windows username, NMB/SMB/NetBIOS/Windows domain name, and master browser. It can discover all the NMB/SMB/NetBIOS/Windows hosts on a local area network by using the hosts lists maintained by master browsers.

You can download the version 1.2.6 of nmbscan tool from HERE.

After downloading, extract the files by typing:

mkdir nmbscan && tar -xvf nmbscan-1.2.6.tar.gz --directory nmbscan

Running nmbscan shows pretty much of information about the usage.

samar@Techgaun:~/Downloads/nmbscan$ ./nmbscan 
nmbscan version 1.2.6 - Techgaun - Sat Mar 31 00:04:15 NPT 2012

usage :
 ./nmbscan -L
  -L show licence agreement (GPL)

 ./nmbscan {-d|-m|-a}
  -d show all domains
  -m show all domains with master browsers
  -a show all domains, master browsers, and hosts

 ./nmbscan {-h|-n} host1 [host2 [...]]
  -h show information on hosts, known by ip name/address
  -n show information on hosts, known by nmb name

You can figure out the command line switches as per your necessity while using the tool. I hope this tool counts as useful for you. :)


Read more...

Automating Execution Of Applications In DosBox

Running dosbox and mounting the necessary directory everytime to run any software in dosbox becomes boring with time. I had to automate the execution of qbasic by using dosbox in edubuntu and so here is the tutorial for making a simple desktop entry for automating the execution of apps in dosbox.

I will be giving an example of qbasic here but you can follow the same method, of course with little modification(that you'll easily figure out). So lets start.

The first thing we will do is make a new configuration file for our qbasic at /opt/qbasic/qbasic.conf. The file will consist of following content and you need to slightly modify according to your path and command name.

[autoexec]
mount C ~/qbasic
C:
qb

Basically what we are doing above is adding our commands in the autoexec section of configuration file that will be read by dosbox. In the autoexec section, we first mount our appropriate directory(~/qbasic as C) and then switch to the mount point and finally execute the required command(qb in above example).

Now all you need to do is create a new launcher with the following command:

/usr/bin/dosbox -conf /opt/qbasic/qbasic.conf

Notice that I'm providing my custom configuration while running the dosbox command. As per your necessity, you could provide -noconsole and -exit switches in the command above(as in games). Also, though this article focusses on linux, you can follow this with minor OS specific variations to run in windows as well. I hope this helps you. :)


Read more...

Extracting All Hyperlinks From Webpages - Python

In this example, I am going to show how easily you can extract all the links in a webpage using python. If you are learning to write some small scale crawler, this can be a quick startup on how you can extract the links in any webpage.

Basically, we will send the http request to any webpage and we will read the HTML response except in the case when the connection can not be established. In such case, we will simply inform the user that we could not connect to the website.

For all these stuffs, we will import few modules and most important ones are re and urllib2 for regular expression stuff and HTTP request/response stuffs respectively.

We then write the regex for the hyperlinks for which we will make a search in the HTML data we get back after sending the request from the server. Note the <a href=[\'"]?([^\'" >]+). The small brackets are there to let us capture our necessary information i.e. the actual links.

Now you understood what we'll be doing, below is the python script to extract the hyperlinks from any webpage.

#!/usr/bin/python

import re, urllib2
from sys import argv

if (len(argv) != 2):
    print "No URL specified. Taking default URL for link extraction"
    url = "http://www.techgaun.com"
else:
    url = str(argv[1])
    
links_regex = re.compile('<a href=[\'"]?([^\'" >]+)', re.IGNORECASE)
url_request = urllib2.Request(url)
try:
    response = urllib2.urlopen(url_request)
    html = response.read()
    links = links_regex.findall(html)
    print '\n'.join(links)
except urllib2.URLError:
    print "Can't Connect to the website"

Now run the script as python extracter.py http://www.techgaun.com or any URL you wish to.

So isn't it a good start for writing your own simple web crawler? :P


Read more...

How To Fix NTFS Disk Partition From Linux

If you have problematic NTFS partition in your hard disk, you can fix many of the common NTFS inconsistencies from linux. Linux consists of a set of tools that allow you to manipulate and perform different types of actions on the NTFS partitions. This package is known as ntfsprogs.

If your linux distribution does not consist of the ntfsprogs package, you can install it by using the package manager tool that comes in your distribution or from command line. Debian and ubuntu users can type the following command:

sudo apt-get install ntfsprogs

Now to fix the NTFS drive, we must first determine the partition we want to fix. We can use the simplest one, the fdisk utility to determine the partition of hard disk we want to fix. Type the following command to view the list of partitions:

sudo fdisk -l

If you have more than one HDDs and want to view partitions of specific HDD, you can always do so by issuing the commands such as sudo fdisk -l /dev/sda or sudo fdisk -l /dev/sdb and so on.

Now lets suppose its /dev/sdb5 we need to fix. We can now use the ntfsfix command that comes in the ntfsprogs package.

sudo ntfsprogs /dev/sdb5

Note that it only repairs some fundamental NTFS inconsistencies, resets the NTFS journal file and schedules an NTFS consistency check for the first boot into Windows. You may run ntfsfix on an NTFS volume if you think it was damaged by Windows or some other way and it cannot be mounted.


Read more...

Thoughts On Combining Compression and Encryption

One of the issues while talking about encryption and cryptography is how should we combine compression with encryption. Data compression is one of the tasks people often do. Combining compression and encryption needs some addressing since compression should always be done before the encryption and not the other way.

The results are generally not good if encryption is done before compressing the data. This is because of the nature of the encryption. Compression takes advantage of non-randomness of data but a good encryption generates the random stream of data which is unlikely to get good compression in cases of loss-less compressions. Of course, some image compression which are not loss-less will still get some compression.

Compression technology looks for the repeatability of data and performs compression by looking such patterns. Most encryption schemes transform the data such that it is random or very very close to being random. Output of good encryption scheme must be indistinguishable from truly random. And compressing the truly random data would not produce effective result. Hence, compress first and then do the encryption. :)


Read more...

How To Copy Text To Clipboard From Command Prompt

I had earlier posted about alternate data streams and the post consisted of texts copied from command line. I was on local IRC channel, one guy was curious if I was using the redirection operator to get the content from the command prompt. So I thought to share this simple tip to copy text from command prompt in windows. Follow the steps as below: 1) Right click anywhere on the command prompt window and then select the Mark option.

2) Now start selecting the text you need to copy using your mouse. You could keep on holding mouse and then do the selection. Alternatively, you could click on the starting point and then while holding the SHIFT key, click on the end of text you wish to copy.

3) After selecting the required text, just press Enter. Alternatively, you can right click on the top title bar of command prompt and then go to Edit -> Copy. If you are looking for copy pasting methods in linux terminals, you can read my article. I hope this helps some of you guys. :)


Read more...