Debug Build Failures on Linux platform

Tools/Unix commands which helps debug build failures

1. nm
2. file
3. objdump
4. strings
5. ar
6. ldd
7. Know about the significance and behavior of different types of symbols.

vim commands/shortcuts

Vim Basics

• Save and Exit (Command mode)

          1. 'w'  : Save the file with latest changes but not exit out of editor

          2. 'wq': Save the file and exit.

          3. 'e'   : Refresh the file if the file has undergone modification from a different process.

                      Note: Any changes to that file from the editor will be lost.

• Navigation keys

          1. 'Ctrl f' : One page forward

          2. 'Ctrl b' : One page backwards

          3. 'w' : Move forward by 1 word

              'W' : Move forward until next space character

          4. 'b' : Move back 1 word

              'B' : Move back to previous space character.

• Search

          1. 'Shift 8' /  '*' : Search the word under cursor

• Increment and Decrement number ( normal/edit mode ):  

          'Ctrl-a' and 'Ctrl-x'

• Add/modify a column of file (vertically)

          Follow the steps:

          1. 'Ctrl-v' : Creates visual block.

          2. Navigate up/down the lines for which you need add/modify. ( Still in visual mode)

          3. 'Shift i'

          4. 'esc'

             

• Windows (commandline mode):

          'vsp <filename>' : Vertical Split
          'sp  <filename>'   : Split window horizontally

          Navigation keys:

           'Ctrl-ww' : Rotate between windows
           'Ctrl-w<arrow keys>' : Navigate to corresponding window as per the arrow key. [ eg: right arrow to move to right window from current line in the current window]

• Tabs (commandline mode):

           'tabe <filename>': Will open the file in a new tab.

          Navigation keys:

           'gt' : Move to next tab. Rotates to first if it's the last tab.
           'gt <number> : Moves to the tab with the given number. [ Ex: 'gt 2' will goto the second tab]

Optimizing file operations efforts using 'vim'. 

Modify multiple files with "similar" operation

This is achieved by recording the operation and recursing that over all the files.

1. Open all the files in vim together. 'vim file1 file2 file3.. file4'  ( If you have a file with list of all the filenames, you can open as: vim `cat file_with_paths` ( notice that back quotes are used here!)
2. Start the first recording: Type 'q' followed by <any character> in command mode.That particular character will be the name of what is being recorded.  Example: 'qs'. All the vim operations you do or commands you run, will start getting recorded now.
3. Do the "similar" operation. Lets consider substituting all the occurrences of  the word "BAD" with "GOOD" as "similar" operation. In command mode, type '%s/BAD/GOOD/g'
4. Save the file and move to the next file: 'wn'  ( 'w' for save, 'n' is to open next file.)
5. Now, run the recording. '@' followed by <any character> will run the recording. In our example, its '@s'. Basically, this particular recording is calling itself! which makes it recursive.
6. End the recording: Type just the single character 'q' to end recording.We have the first recording encapsulated inside itself along with the commands which need to be run for all the files.
7. Run the recording to finish: Type '@s'. Now, all the files having the word 'BAD' will have replaced with the word 'GOOD', in a jiffy :)

Vim Sessions

Create session

:mksession ~/mysession.vim

source that vim file to get the old session

:source ~/mysession.vim

$ vim -S ~/mysession.vim

Amazon aws - view ubuntu desktop using VNC

Setting up VNC on Ubuntu in Amazon EC2
Install the desktop packages and remote desktop (VNC) server:

sudo apt-get update
sudo apt-get install ubuntu-desktop
sudo apt-get install vnc4server

Configure the VNC server enter vncserver, which will then prompt you to create a password. Once created, entervncserver -kill :1 to stop the server.

edit the config file: ~/.vnc/xstartup.

Uncomment the two lines below the string "Uncomment the following two lines for normal desktop." And on the second line add "sh" so the line reads exec sh /etc/X11/xinit/xinitrc. 

Start the VNC server again by entering vncserver. 

Open VNC through Tunneling putty :

References:
http://crl.ucsd.edu/handbook/vnc/

Filtering django admin inline dropdown for a foreignkey based on the parent

The original concept is explained here. But overriding the method 'formfield_for_dbfield' as in that link doesn't show up the column name. To avoid that override the 'formfield_for_foreignkey' with same logic but just build the queryset as necesssary.

Following is the snippet from my code:

class NewsBentInline(admin.TabularInline):
    model = NewsBent
           
    def __init__(self, model, admin_site, dist=False):
        self.dist = dist
        super(NewsBentInline, self).__init__(model, admin_site)
           
    def formfield_for_foreignkey(self, field, request, **kwargs):
        parent_article = self.get_object(request, Article)
        queryset = None
        if field.name == "parliament":
            if parent_article.district == 'District_name1':
                queryset = Parliament.objects.filter(dist__name="District_name1")
            else:
                queryset = Parliament.objects.all()
        if queryset is not None:
            kwargs["queryset"] = queryset
        return super(NewsBentInline, self).formfield_for_foreignkey(field, request, **kwargs)

    def get_object(self, request, model):

        object_id = request.META['PATH_INFO'].strip('/').split('/')[-1]

        return model.objects.get(pk=object_id)

Kibana on ubuntu linux

Install Java:
sudo apt-get install openjdk-7-jre

Download logstash from http://www.elasticsearch.org/overview/logstash/.

It comes bundled with elasticsearch and kibana.

Setup/Installation instructions for crawler - scrapy for scraping pages and Django for frontend

Requirement:

Ubuntu 12.04 LTS.

Python version: 2.7.3

Setup:

sudo apt-get install python-pip

sudo pip install virtualenvwrapper

copy the following to .bashrc:

"""

export WORKON_HOME=$HOME/.virtualenvs

export PROJECT_HOME=$HOME/projects

source /usr/local/bin/virtualenvwrapper.sh

"""

mkvirtualenv crawler

workon crawler

Dependent Packages Installation:

sudo apt-get install libxml2-dev

sudo apt-get install libxslt-dev

sudo apt-get install python2.7-dev

sudo apt-get install python-scrapy

sudo apt-get install libffi-dev

pip install Scrapy

If the above packages are already available in global site packages you can use them by running following virtual env command, more info here:
toggleglobalsitepackages

mysql:

sudo apt-get install python-mysqldb

sudo apt-get install mysql-client-core-5.5

sudo apt-get install mysql-server-core-5.5

sudo apt-get install mysql-server

pip install SQLAlchemy

Django:

pip install Django==1.6.1

BeautifulSoup:

pip install beautifulsoup4

NodeJS for running javascript on terminal/commandline:
http://nodejs.org/download/

Set the environment variable:
Be careful about the ordering of paths, virtual env should be followed by system installed python
export PYTHONPATH="/home/vjonnak/.virtualenvs/crawler/local/lib/python2.7/site-packages:/usr/lib/python2.7/dist-packages"

Create the DB to support utf8 encoding:

create database news_db  DEFAULT CHARACTER SET utf8   DEFAULT COLLATE utf8_general_ci;

Adjust resolution of Oracle VM virtualbox with ubuntu

To adjust Screen resolution:

Click on 'Devices' on the virtualbox menu -> 'Insert Guest additions CD Image'
Login to Ubuntu:
Devices -> VBOXADDITIONS
On the top right end of explorer click the button "Open Autorun prompt"
Restart Ubuntu

Extend/Reuse the power of Django Admin model

urls.py:

"""
urlpatterns = patterns('',
  url(r'articleview/$', 'myapp.views.admin_reuse'),
  url(r'^admin/', include(admin.site.urls)),
)
"""

admin.py:

"""
from myapp.models import Article

class ArticleAdmin(admin.ModelAdmin):
    <...>

admin.site.register(Article, ArticleAdmin)

"""

views.py:

"""
from django.contrib.admin.sites import AdminSite

from myapp.models import Keyword, Article

from myapp.admin import ArticleAdmin

def admin_reuse(request):

   article_admin = ArticleAdmin(Article, AdminSite(), dist)

   return article_admin.changelist_view(request)

"""

UnicodeEncodeError exception in python

Error message:
UnicodeEncodeError: codec can't encode character  in position : ordinal not in range(128)

Python represents the string in utf-8 though the string has charachters from different encoding, hence the error message.

Scenario: Have charachters in string which are represented as string(utf-8) but has to be unicode

• Check the type, that should tell what's the encoding:

           type(str_var)

• If the type is 'str': you should be decoding from utf-8 to unicode:

           str_var.decode('utf-8')

• If the type is unicode, then encode it to utf-8

           univar.encode('utf-8')

Note: if the encoding is wrong you might end up with no value if you do to avoid the exception:
          str_val.encode('ascii', 'ignore')

Refer unicode link to understand more about unicode.

Unix/Linux Shell commands

Unix/Linux Shell commands

Shortcuts

!!  (bang bang): Run entire previous command line.

    Previous command:
        $ ls -l /usr/bin/g++ /usr/bin/gcc
          -rwxr-xr-x 4 root root 221736 Oct  7  2010 /usr/bin/g++
          -rwxr-xr-x 2 root root 221920 Oct  7  2010 /usr/bin/gcc
     Shortcut:
        $ !!
           ls -l /usr/bin/g++ /usr/bin/gcc
           -rwxr-xr-x 4 root root 221736 Oct  7  2010 /usr/bin/g++
           -rwxr-xr-x 2 root root 221920 Oct  7  2010 /usr/bin/gcc

!^  (bang caret): Run just the command/program from previous command line

     Previous command:
         $ echo "contents of file1" > file1 ; echo "contents of file2" > file2

         $ ls file1 file2

            file1  file2

      Shortcut:

         $ cat !^
            cat file1
            contents of file1

!* (bang star): All arguments of previous command line

    Previous command:
         $ echo "contents of file1" > file1 ; echo "contents of file2" > file2

         $ ls file1 file2

            file1  file2
    Shortcut:
         $ head !*
            head file1 file2
            ==> file1 <==
            contents of file1

            ==> file2 <==
            contents of file2

Navigating/edit command line on terminal

Ctrl + A or Home	Moves the cursor to the start of a line.
Ctrl+ E or End	Moves the cursor to the end of a line.
Esc + B	Moves to the beginning of the previous or current word.
Ctrl + K	Deletes from the current cursor position to the end of the line.
Ctrl + U	Deletes from the start of the line to the current cursor position.
Ctrl + W	Deletes the word before the cursor.
Alt + B	Goes back one word at a time.
Alt + F	Moves forward one word at a time.
Alt + C	Capitalizes letter where cursor is and moves to end of word.

Commands

ls : list directory contents

          1. List the files by directories but not recurse with -l (minus l) flag
              ls -ld *

          2. Find timestamp of a file in seconds:
              ls -l --time-style=full-iso

Copy files across sites/machines

scp <machine name>:<source file path>  <machine name>:<Destination path>
Note: Machine name is not necessary if you are already logged into that machine

reset: Reset the terminal

• Clears the garbled terminal with illegible characters 

locale: Displays the current locale settings

More Unix tricks here
http://lifehacker.com/5743814/become-a-command-line-ninja-with-these-time-saving-shortcuts
https://help.ubuntu.com/community/UsingTheTerminal

Settings

setenv LC_ALL C

If there are few illegible characters displayed in place of special charachters in files or man pages set the above to make them visible properly.

Learn locale here, here