UbuntuMongol Blogs

Using Python and Shred to securely remove directories and files on a Linux File System - Updated

2011-04-24T18:18:00.000-07:00

Creating an android app on an Ubuntu Platform with android SDK and Google's Python Android scripting

2011-02-12T00:53:00.000-08:00

Creating shell code on Ubuntu 10.04 using the chmod system call Part 1

2010-07-31T23:47:00.000-07:00

I have decided to document some basic introductory information concerning creating NASM and C code on my Ubuntu build.-

This is just my documentation space on how to create shell code on a linux machine. The syscall that I have chosen for this project is chmod and "testfile" at its target.

This particular example will change the permission of the "testfile" to 777 in the same directory.

Part 2 of this exercise will probably change the permission on the shadow file using a vulnerable program. This should not be to difficult since I already have the asm code and just need to specify a file path and choose my vulnerable program.

Step 1 -

Ensure that you have nasm and g++ on your ubuntu build

$sudo apt-get install nasm

Step 2

Create an assembly program call ch_test.asm

$ touch ch_test.asm

$ gedit ch_test.asm

;ch_tests.asm ; File name of assembly program

[section .text]

global _start

_start: ; Initiate my global start and jump command

jmp short ender ; Jump to the ender place holder in code

starter: ; return from previous jump this helped store file location

xor ecx,ecx ; clear memory (zeroize)

xor ebx,ebx

xor eax,eax

pop ebx ; I popped this to ensure we

mov al, 15 ; reduce Null by using 8-bit this is chmod system call

mov ecx, 511 ; This is the octal rep for 777 on Linux

int 80h ; Execute

xor ebx, ebx ; Clear memory for ebx

mov eax, 1 ; Syscall for exit

int 80h ; Execute

ender: ;ender instruction location of my file

call starter

db 'testfile' ;test file name in the same directory

Step 3- test my nasm code.

$ nasm -f elf ch_test.asm #This will create an ch_test.o file

$ ld -o ch_test ch_test.o

$ strace ./ch_test

execve("./ch_test", ["./ch_test"], [/* 37 vars */]) = 0
chmod("testfile", 0777) = 0
_exit(0) = ?

$ objdump -d ch_test # The following output will be displayed,

ch_test:     file format elf32-i386

Disassembly of section .text:

08048060 <_start>:
8048060:   eb 19                   jmp    804807b

08048062 :
8048062:   31 c9                   xor    %ecx,%ecx
8048064:   31 db                   xor    %ebx,%ebx
8048066:   31 c0                   xor    %eax,%eax
8048068:   5b                      pop    %ebx
8048069:   b0 0f                   mov    $0xf,%al
804806b:   b9 ff 01 00 00          mov    $0x1ff,%ecx
8048070:   cd 80                   int    $0x80
8048072:   31 db                   xor    %ebx,%ebx
8048074:   b8 01 00 00 00          mov    $0x1,%eax
8048079:   cd 80                   int    $0x80

0804807b :
804807b:   e8 e2 ff ff ff          call   8048062
8048080:   74 65                   je     80480e7
8048082:   73 74                   jae    80480f8
8048084:   66                      data16
8048085:   69                      .byte 0x69
8048086:   6c                      insb   (%dx),%es:(%edi)
8048087:   65                      gs

Step 4- use the created nasm objdump information.

$ touch ch_test.c

$ gedit ch_test.c

# This values are obtained from the objdump form the asm .

const char code[]= "\xeb\x19\x31\xc9\x31\xdb\x31\xc0\x5b\xb0\x0f\xb9\xff\x01\x00\x00\xcd\x80\x31\xdb\xb8\x01\x00\x00\x00\xcd\x80\xe8\xe2\xff\xff\xff\x74\x65\x73\x74\x66\x69\x6c\x65";

int main(int argc, char **argv)

{

int (*func)();

func = (int (*)()) code;

(int)(*func)();

}

Then you want to compile the c program to make it executable.

$ g++ -g ch_test.c -o ch_test

Execute the program

$ strace ./ch_test

Perform some gdb on the binary to see the memory information.

$ gdb -q ./ch_test

(gdb) list

2 const char code[]= "\xeb\x19\x31\xc9\x31\xdb\x31\xc0\x5b\xb0\x0f\xb9\xff\x01\x00\x00\xcd\x80\x31\xdb\xb8\x01\x00\x00\x00\xcd\x80\xe8\xe2\xff\xff\xff\x74\x65\x73\x74\x66\x69\x6c\x65";

5 int main(int argc, char **argv)

6 {

7 int (*func)();

8 func = (int (*)()) code;

9 (int)(*func)();

10 }

(gdb) disas main

Dump of assembler code for function main:

0x08048494 <+0>: push %ebp

0x08048495 <+1>: mov %esp,%ebp

0x08048497 <+3>: and $0xfffffff0,%esp

0x0804849a <+6>: sub $0x10,%esp

0x0804849d <+9>: movl $0x80485a0,0xc(%esp)

0x080484a5 <+17>: mov 0xc(%esp),%eax

0x080484a9 <+21>: call *%eax

0x080484ab <+23>: mov $0x0,%eax

0x080484b0 <+28>: leave

0x080484b1 <+29>: ret

End of assembler dump.

(gdb) break main

Breakpoint 1 at 0x804849d: file ch_test.c, line 8..

(gdb) run

Starting program: /home/t/coding/t_coding/ch_test

Breakpoint 1, main (argc=1, argv=0xbffff4f4) at ch_test.c:8

8 func = (int (*)()) code;

(gdb) info registers

eax (Accumulator) 0xbffff4f4 -1073744652

ecx (Counter) 0x758d8dac 1972211116

edx (Data) 0x1 1

ebx (Base Register) 0x283ff4 2637812

esp (Stack Pointer) 0xbffff430 0xbffff430

ebp (Base Pointer) 0xbffff448 0xbffff448

esi (Source Index) 0x0 0

edi (Destination Index) 0x0 0

eip (Current Instruction) 0x80483bd 0x80483bd

eflags 0x286 [ PF SF IF ]

cs 0x73 115

ss 0x7b 123

ds 0x7b 123

es 0x7b 123

fs 0x0 0

gs 0x33 51

(gdb) info registers eip or below

eip 0x804849d 0x804849d

(gdb) i r eip

eip 0x804849d 0x804849d

(gdb) x/o $eip # show in octal

0x80483bd : 01411042307

(gdb) x/x # show in hexadecimal

0x80483c1 : 0x08048490

(gdb) x/8xb # show the memory in 8 hexadecimal bytes

0x80483c5 : 0x8b 0x44 0x24 0x0c 0xff 0xd0 0xc9 0xc38

(gdb) x/8xh # half word 2bytes or x/8xw #word 4bytes or x/8xg 8bytes

0x80483cd: 0x9090 0x5590 0xe589 0xc35d 0x748d 0x0026 0xbc8d 0x0027

Notes about – dump and gdb

The first three lines have to do with stack preperation -

The lines that need to be analyzed are movl and mov -

info registers – displays the contents of integer registers -

Help info will give you some additional commands in gdb

EAX, EBX, ECX, and EDX 32 Bit

AH, BH, CH and DH 16 Bit

AL, BL, CL, and DL 8 Bit

VMware or VirtualBox on Ubuntu 10.04 - installation and conversion

2010-07-17T14:18:00.000-07:00

It seems like whenever I perform an upgrade as an example from 9.10 to 10.04, the VMware build seems to lose it functionality. This has caused me to utilize VirtualBox some months ago and the first key item that I noticed was the ease of installation.

INSTALLING VirtualBox on Ubuntu

So simple,

They fully support debian and more particularity Ubuntu 10.04 ("Lucid Lynx") with a *.deb download -

http://www.virtualbox.org/wiki/Linux_Downloads

So you basically just click on the link and use the package manger to run the GDebi package installer -

I did not take any notes when I installed the app but remember it was very simple and I did not have to load any additional libraries - but since I did not take notes you might have to load some dependencies - sorry I just don't remember.

INSTALLING VMware on Ubuntu

This section is a pain - As far as I know and based on some searching there is not a supported versions specifically built for ubuntu 10.04 so you have to download the VMware-Player*.bundle file manually to install the application.

These day - were Time is so so premium and if it takes more time that it should to perform a function or use a specific environment, then I turn to other tested solutions.

This was my quick experience trying to install VMware on Ubuntu -

I found some documentation were you download the VMware-Player*.bundle from the website after you register.

Then you change the permission's to +x and try and run the bundle.
$ chmod +x VMware-Player*.bundle chmod +x VMware-Player*.bundle
$ gksudo bash ./VMware-Player*.bundle

This resulted in a whole bunch of errors and yes I have g++ installed -

so I tried

$ sudo ./VMware-Player-3.0.0-203739.i386.bundle --ignore-errors

Then I received this message -

./VMware-Player-3.0.0-203739.i386.bundle: 110: Syntax error: newline unexpected

As stated before I am at a turning point in my life were I used to spend the hours trying to troubleshoot this error message. So if I am aware of a comparable product that can produce the same results that has me up and running in minutes compared to an hour of trouble shooting then I am sold.

So I gave up on VMware on the Ubuntu platform and I have been currently using VirtualBox and will continue to do so,

I can say in VMware defense that at my current workplace were I am forced to use Windows in some areas, the VMWare Workstation and Server products are great.

This is my way of letting of some steam and as a quick reference for a later date so I can remind myself why I am using VirtualBox -

Installation of Metasploit Framework on Ubuntu Linux

2010-07-10T15:22:00.000-07:00

I have a new ubuntu build (10.04), and I am installing all of my usual software and of course I have to re-install metasploit for my internal security lab testing purposes.

I am referencing the below link for the installation and this blog as my personal documentation space so I do not have to re-invent the wheel on all future installations.

So far the documentation has been sound with no hiccups - I am currently installing the tarball and will perform some testing.

I also plan on writing some ruby code and document the process.

Basic Metasploit Setup on UBUNTU -
http://www.metasploit.com/redmine/projects/framework/wiki/Install_Ubuntu

The database that I have selected in this installation is -

$ sudo apt-get install rubygems libmysqlclient-dev
$ sudo gem install mysql

I have some experience with MySQL and I image that I will utilize the db in future programming projects on this server.

Also ensure that you have mysql-server installed

$ apt-get install mysql-server

--------------------------- MySQL user setup -------------------------
Since this was a new ubuntu build - I did have to install MySQL and created a user for metasploit -

mysql>create user 'useraname'@'localhost' identified by 'password';

mysql>GRANT SELECT,INSERT,UPDATE,DELETE ON *.* TO 'userame'@'localhost';

mysql>GRANT ALL ON *.* TO 'username'@'localhost';

-----------------------------------------------------------------------------

All sections of the installation have been completed except the "To enable WiFi modules:"

Now the fun begins starting Metasploit and testing the functionality.

In this install the correct full path to start is -

Start Metasploit

$ /opt/metasploit3/msf3/msfconsole

Basic Metasploit commands to get started

1.Ensure that you can connect to your preferred database which in my case is MySQL -

msf > db_connect username:password@locahost/metasploit3

#Note this will create a metasploit db in your mysql-server on localhost -

2. Metasploit does have a ? help command and will allow you to choose from the core commands and Database Backend Command -

Core Commands
=============

    Command       Description
    -------       -----------
    ?             Help menu
    back          Move back from the current context
    banner        Display an awesome metasploit banner
    cd            Change the current working directory
    color         Toggle color
    connect       Communicate with a host
    exit          Exit the console
    help          Help menu
    info          Displays information about one or more module
    irb           Drop into irb scripting mode
    jobs          Displays and manages jobs
    kill          kill a job
    load          Load a framework plugin
    loadpath      Searches for and loads modules from a path
    quit          Exit the console
    resource      Run the commands stored in a file
    route         Route traffic through a session
    save          Saves the active datastores
    search        Searches module names and descriptions
    sessions      Dump session listings and display information about sessions
    set           Sets a variable to a value
    setg          Sets a global variable to a value
    show          Displays modules of a given type, or all modules
    sleep         Do nothing for the specified number of seconds
    unload        Unload a framework plugin
    unset         Unsets one or more variables
    unsetg        Unsets one or more global variables
    use           Selects a module by name
    version       Show the framework and console library version numbers

Database Backend Commands
=========================

    Command               Description
    -------               -----------
    db_add_host           Add one or more hosts to the database
    db_add_note           Add a note to host
    db_add_port           Add a port to host
    db_autopwn            Automatically exploit everything
    db_connect            Connect to an existing database
    db_create             Create a brand new database
    db_del_host           Delete one or more hosts from the database
    db_del_port           Delete one port from the database
    db_destroy            Drop an existing database
    db_disconnect         Disconnect from the current database instance
    db_driver             Specify a database driver
    db_hosts              List all hosts in the database
    db_import             Import a scan result file (filetype will be auto-detected)
    db_import_amap_log    Import a THC-Amap scan results file (-o )
    db_import_amap_mlog   Import a THC-Amap scan results file (-o -m)
    db_import_ip_list     Import a list of line seperated IPs
    db_import_msfe_xml    Import a Metasploit Express report (XML)
    db_import_nessus_nbe Import a Nessus scan result file (NBE)
    db_import_nessus_xml Import a Nessus scan result file (NESSUS)
    db_import_nmap_xml    Import a Nmap scan results file (-oX)
    db_import_qualys_xml Import a Qualys scan results file (XML)
    db_nmap               Executes nmap and records the output automatically
    db_notes              List all notes in the database
    db_services           List all services in the database
    db_status             Show the current database status
    db_sync               Synchronize the database
    db_vulns              List all vulnerabilities in the database
    db_workspace          Switch between database workspaces

3. The db_autopwn is the easiest to use and will show you the options that are associated with this command at the command prompt -

Usage: db_autopwn [options]
    -h          Display this help text
    -t          Show all matching exploit modules
    -x          Select modules based on vulnerability references
    -p          Select modules based on open ports
    -e          Launch exploits against all matched targets
    -r          Use a reverse conndbect shell
    -b          Use a bind shell on a random port (default)
    -q          Disable exploit module output
    -R [rank] Only run modules with a minimal rank
    -I [range] Only exploit hosts inside this range
    -X [range] Always exclude hosts inside this range
    -PI [range] Only exploit hosts with these ports open
    -PX [range] Always exclude hosts with these ports open
    -m [regex] Only run modules whose name matches the regex
    -T [secs] Maximum runtime for any exploit in seconds

4. I added a single host to test the functionality of this build -

msf > db_add_host xxx.xxx.xxx.xxx

msf > db_hosts # This will ensure you are using the correct hosts in the databases.

5. You can also use nmap to search out hosts and port - # on this build I had to install nmap

------------------------- Install Nmap on Ubuntu------------------------------

$ sudo apt-get install nmap

-------------------------- Install Nmap on Ubuntu------------------------------

msf > db_nmap -sS -T4 -O x.x.x.0/24
or db_nmap -sS -T4 -O X.X.X.X

NMAP switches -
-sS/sT/sA/sW/sM: TCP SYN/Connect()/ACK/Window/Maimon scans
-T<0-5>: Set timing template (higher is faster)
OS DETECTION:
-O: Enable OS detection

6. Once you build your database of known hosts and ports you can then choose the lazy way

msf > db_autopwn -t -p -e -s -b   # See step three for switch defs.

Dirk Loss: Python tools for penetration testers

2010-07-05T20:54:00.000-07:00

Dirk Loss: Python tools for penetration testers

CISSP Nationwide Average Salary & PayScale

2010-06-13T19:39:00.000-07:00

CISSP $96,000

View Larger Salary Graph

PayScale

cissp Job Trends

Cissp jobs

Converts video files to mp4

2010-06-13T11:09:00.000-07:00

Here is a quick and dirty post on converting video files to mp4 using a perl script that I wrote and the mp4size ruby script written by Thomer M. Gil

---
------------------------------------------------------------------------------------------------------------------------

#! /usr/bin/perl -wl
# This script will take multiple files in a directory and automatically convert to #.mp4 using .mp4size script -
# Ensure that this ruby and perl script is in the same directory were the files #need to reside
# Written by Tim E
# twitter @ubuntumongol

use strict;
use warnings;

my$file;
my $dir = "/PathtoFiles/";
my $newdir ;
opendir(BIN, $dir) or die "Can't open $dir: $!";
my $newfile;
my $origfile;

while( defined ($file = readdir BIN) ) {
chomp ($file);
my $filex;
my @parts = split (/\./,$file);
$filex = $parts[1];
pop @parts;
my $file_no_ext = join '.', @parts;

$newfile = "\"".$dir.$file_no_ext . ".mp4"."\"";
$origfile = "\"".$dir.$file."\"";
$newdir = "\"".$dir."mp4ize"."\"";
#my $runcommand = "ffmpeg -i $origfile -target ntsc-vcd $newfile";

my $runcommand = "$newdir $origfile";
print "$runcommand\n";

if ($filex eq "mpg")
{
system($runcommand);

}

}
closedir(BIN);

---------------------------------------------------------mp4ize-------------------------------------------------------------------

#!/usr/bin/ruby -w
#
# Copyright (C) 2007-2009 Thomer M. Gil [http://thomer.com/]
#
# Thanks to Brian Moore, Justin Payne, Matt Spitz, Martyn Parker,
# Jean-Francois Macaud, Thomas Hannigan, Anisse Astier, Juanma Hernández,
# Trung Huynh, and Mark Ryan for bugfixes and suggestions.
#
# Oct. 14, 2008: show percentage progress. add -t and -w flags.
# Jan. 11, 2009: switch to bit/s bitrates for newer ffmpeg versions.
#                add --iphone option.
#                add -y option to ffmpeg (overwrite).
# Jan. 20, 2009: don't exit early when processing multiple files.
# Feb. 17, 2009: deal with "Invalid pixel aspect ratio" error.
# Apr. 1, 2009: new --outdir parameter.
# May 22, 2009: handle filenames with quotes and whitespace.
# Oct   6, 2009: fix bug where we forget to read stderr
# Nov. 5, 2009: fix -v, -t, and -w command line options
#                removed bogus 'here' debug statement
#
# This program is free software. You may distribute it under the terms of
# the GNU General Public License as published by the Free Software
# Foundation, version 2.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# This program converts video files to mp4, suitable to be played on an iPod
# or an iPhone. It is careful about maintaining the proper aspect ratio.
#

require 'getoptlong'
require 'open3'

# will automatically try with -vcoded libxvid, also.
# will automatically try with -acodec libfaac, also.
DEFAULT_ARGS = "-f mp4 -y -vcodec xvid -maxrate 1000 -qmin 3 -qmax 5 -g 300 -acodec aac"
DEFAULT_BUFSIZE = 4096
DEFAULT_AUDIO_BITRATE = 128 # will be automatically multiplied with 1024 for newer ffmpeg versions
DEFAULT_VIDEO_BITRATE = 400 # will be automatically multiplied with 1024 for newer ffmpeg versions
IPOD_WIDTH = 320.0
IPOD_HEIGHT = 240.0
IPHONE_WIDTH = 480.0
IPHONE_HEIGHT = 320.0

$options = {}
opts = GetoptLong.new(*[
[ "--audio", "-a", GetoptLong::REQUIRED_ARGUMENT ], # audio bitrate
[ "--help", "-h", GetoptLong::NO_ARGUMENT ],         # help
[ "--video", "-b", GetoptLong::REQUIRED_ARGUMENT ], # video bitrate
[ "--verbose", "-v", GetoptLong::NO_ARGUMENT ],      # verbose
[ "--width", "-w", GetoptLong::REQUIRED_ARGUMENT ], # override width
[ "--height", "-t", GetoptLong::REQUIRED_ARGUMENT ], # override height
[ "--iphone", "-i", GetoptLong::NO_ARGUMENT ],       # set width/height
[ "--outdir", "-o", GetoptLong::REQUIRED_ARGUMENT ], # dir where to write files
])
opts.each { |opt, arg| $options[opt] = arg }

if $options['--help']
puts <
mp4ize - encode videos to mp4 for an iPod or an iPhone

Usage: mp4ize file1.avi [file2.mpg [file3.asf [...]]]

Options:

-h/--help          : this help
-v/--verbose       : verbose

-a/--audio RATE    : override default audio bitrate (#{DEFAULT_AUDIO_BITRATE})
-b/--video RATE    : override default video bitrate (#{DEFAULT_VIDEO_BITRATE})

-w/--width WIDTH   : over default width (#{IPOD_WIDTH.to_i})
-t/--height HEIGHT : over default height (#{IPOD_HEIGHT.to_i})
-i/--iphone        : same as --width #{IPHONE_WIDTH.to_i} --height #{IPHONE_HEIGHT.to_i}

-o/--outdir O      : write files to given directory
EOF
exit
end

# --iphone sets --width and --height
if $options['--iphone']
if $options['--width'] || $options['--height']
    warn "You can't use --iphone with --width or --height."
    exit 1
else
    $options['--width'] = $options['-w'] = IPHONE_WIDTH
    $options['--height'] = $options['-t'] = IPHONE_HEIGHT
end
end

audio_bitrate = $options['--audio'] || DEFAULT_AUDIO_BITRATE
video_bitrate = $options['--video'] || DEFAULT_VIDEO_BITRATE

ARGV.each do |infile|
outfile = infile.dup
ext = File.extname(outfile)
outfile.sub!(/#{ext}$/, '.mp4')
if $options['--outdir']
    if !File.directory?($options['--outdir'])
      warn "#{$options['--outdir']} does not exist or is not a directory. exiting."
      exit 1
    end
    outfile = File.join($options['--outdir'], File.basename(outfile))
end

# open the file to figure out the aspect ratio
duration, w, h = 0.0, nil, nil
Open3.popen3("/usr/bin/ffmpeg", "-i", infile) do |stdin, stdout, stderr|
    [stdout, stderr].each do |io|
      io.each_line do |line|
        if line.match(/Video:.+ (\d+)x(\d+)/)
          w, h = $1.to_f, $2.to_f
        elsif line.match(/Duration:\s+(\d+):(\d+):(\d+)\.(\d+)/)
          duration += $1.to_f * 3600
          duration += $2.to_f * 60
          duration += $3.to_f
          duration += $4.to_f / 10
        end
      end
    end
end

begin
    aspect = w/h
rescue
    puts "Couldn't figure out aspect ratio."
    exit
end

user_width = $options['--width'] ? $options['--width'].to_i : IPOD_WIDTH
user_height = $options['--height'] ? $options['--height'].to_i : IPOD_HEIGHT

width = user_width.to_i
height = (width / aspect.to_f).to_i
height -= (height % 2)
pad = ((user_height - height.to_f) / 2.0).to_i
pad -= (pad % 2)
padarg1, padarg2 = "padtop", "padbottom"

# recalculate using the height as the baseline rather than the width
if pad < 0
    height = user_height.to_i
    width = (height * aspect.to_f).to_i
    width -= (width % 2)
    pad = ((user_width - width.to_f)/2.0).to_i
    pad -= (pad % 2)
    padarg1, padarg2 = "padleft", "padright"
end

File.unlink(outfile) if File.exists?(outfile)

# use %infile% and %outfile% and replace those after the split() so that we
# don't split() a filename that has spaces in it.
cmd = "/usr/bin/ffmpeg -i %infile% #{DEFAULT_ARGS} -bufsize #{DEFAULT_BUFSIZE} -s #{width}x#{height} -#{padarg1} #{pad} -#{padarg2} #{pad} -ab #{audio_bitrate} -b #{video_bitrate} %outfile%"
puts cmd if $options['--verbose']

# We could just call "system cmd" here, but we want the exit code of mp4ize
# to tell us whether the duration of the generated mp4 equals the duration
# of the original movie. Exits with a non-zero code if the two are not
# within 1% of each other.

time = 0
STDOUT.sync = true

# try with -vcodec libxvid and -vcodec xvid
# try with -acodec libfaac and -acodec aac
catch(:done) do
    5.times do
      catch(:retry) do
        puts "cmdline: #{cmd}" if $options['--verbose']
        cmd_array = cmd.split(/\s+/)
        cmd_array.collect! {|s| s.sub(/^%infile%$/, infile)}
        cmd_array.collect! {|s| s.sub(/^%outfile%$/, outfile)}
        Open3.popen3(*cmd_array) do |stdin, stdout, stderr|
          io = select([stdout, stderr], nil, nil, 10)
          2.times do |std| # both stdout and stderr
            next if io[0][std].nil?
            io[0][std].each_line("\r") do |line|
              puts "the line is #{line}"
              printf("\r%.2f%% | ", time / duration * 100.0)
              print line
              if line.match(/Invalid pixel aspect ratio/)
                cmd.sub!("-s #{width}x#{height}", "-s #{width}x#{height} -aspect #{aspect}")
                throw :retry
              elsif line.match(/Unknown.*code.*xvid/)
                cmd.sub!('-vcodec xvid', '-vcodec libxvid')
                throw :retry
              elsif line.match(/Unknown.*code.*aac/)
                cmd.sub!('-acodec aac', '-acodec libfaac')
                throw :retry
              # newer ffmpeg versions want bit/s, not kbit/s.
              elsif line.match(/The bitrate parameter is set too low/)
                cmd.sub!(/-ab \d+/, "-ab #{audio_bitrate}k")
                cmd.sub!(/-b \d+/, "-b #{video_bitrate}k")
                cmd.sub!(/-bufsize \d+/, "-bufsize #{DEFAULT_BUFSIZE}k")
                throw :retry
              elsif line.match(/time=([^\s]+)/)
                time = $1.to_f
              end
            end
          end
        end
        throw :done
      end
    end
end

# return completeness of mp4 file
puts "expected duration: #{duration}" if $options['--verbose']
puts "encoded duration: #{time}" if $options['--verbose']
if ARGV.size == 1
    exit((time <= duration * 1.01) && (time >= duration * 0.99))
end
end

Installing MongoDb and Python with eclipse on Ubuntu 9.10

2010-06-12T15:28:00.000-07:00

Installing MongoDb and Python with eclipse on Ubuntu 9.10

System = Ubuntu 9.10

Starting Reference = http://giantflyingsaucer.com/blog/?p=839

I just wanted to document a informative and procedural process in setting up python and mongodb and using the eclipse programming environment, it will also be used to reference the python programming project to organize my ebooks and documentation.

I normally will use MySQL for any projects that require a database and have been using MySQL along with the other commercial databases for some years now. But after reading the Linux Journal and learning that MongoDB is a NoSQL database, this has intrigued my interest and since normal databases are not particularly in my opinion suited to store documents – this has led me to start this personal project.

MongoDB Setup Steps

1. Install the curl application to pull the latest db version from the website.

$ apt-get install curl

2. Once curl is installed on the system – create a folder on your local system to serve and the database repository.

$ mkdir /home/t/data/db

3. Once the directory and created then download the latest build from mongodb

$ curl -O http://downloads.mongodb.org/linux/mongodb-linux-i686-latest.tgz

4. Then untar the download from the website

$ tar xzf mongodb-linux-i386-latest.tgz

5. The files will now reside in the /data/db path to start using mongo db simply start the data bast engine by

$ /data/db/mongodb-linux-i686-2010-05-10/bin/mongod &

6. Open another command terminal and execute the following command to get the mongo db interface

$ /data/db/mongodb-linux-i686-2010-05-01/bin/mongo

You will then get an interesting display of text using this version of mongo – so I guess this is the wrong db but the size it good for now
Sat May 1 13:02:23 Mongo DB : starting : pid = 2597 port = 27017 dbpath = /data/db/ master = 0 slave = 0 32-bit
****
WARNING: This is development version of MongoDB. Not recommended for production.
****
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
** see http://blog.mongodb.org/post/137788967/32-bit-limitations for more
At the command prompt of > will be given and from a security perspective unlike MySQL no password is required and I am unsure of the permission the default user is authorized. I will investigate this further after I get through the initial process

The help feature is a strong point for the mongodb – its great to get commands from the system without having to look them up.

7.>help

The help command returned the following results

show dbs show database names
show collections show collections in current database
show users show users in current database
show profile show most recent system.profile entries with time >= 1ms

db.help() help on DB methods
db.foo.help() help on collection methods
db.foo.find() list objects in collection foo
db.foo.find( { a : 1 } ) list objects in foo where a == 1

I used the web reference example about to create a test db entry and return the results

db.mystorage.save( { "message":"Hello World"} )
db.mystorage.findOne()
{ "_id" : ObjectId("4bdc88f8739846d101246a42"), "message" : "Hello World" }

I also added a little more testing and entered two more entries to mystorage database –

db.mystorage.save( { "message1":"Hello World"} )
db.mystorage.save( { "message2":"Hello World"} )

At this point I used the previously mentioned help function and entered the following command to see my entries within the mystorage db that is apparently created by default,

db.mystorage.find()

{ "_id" : ObjectId("4bdc88f8739846d101246a42"), "message" : "Hello World" }
{ "_id" : ObjectId("4bdc8a1c739846d101246a43"), "message1" : "Hello World" }
{ "_id" : ObjectId("4bdc8a21739846d101246a44"), "message2" : "Hello World" }

------------------------------------------------------------------------------------------------

This was part one of my process and now that I know the db portion is working on home development system it is now time to install python and start the programming process of accessing this database and writing an interface for storing and retrieving documents.

I have utilized C, C++, PHP, and Perl as my primary languages of choice and I am most recently refreshed with Perl, but after some reading on the Internet and some language comparison. I decided to give Python a chance for a couple of reasons.

1. Easier to read by a human compared to the cryptic nature of Perl.

2. I am not really creating a web programming structure so PHP is out the question.
3. I am not concerned with speed for this project and even though C++ is probably my favorite language it would be an overkill for this simple adventure.

So I have a new language to learn that has been mentioned to be more secure and easy for someone with Perl and C++ programming experience.

I will try and define some differences and provide an overall transition assessment of using a language that I have never used before in my previous experiences.

------------------------------------------------------------------------------------------------

Installing Python on my Ubuntu system.

The will start by installing the python-setuptools using the following commands

sudo apt-get install python-setuptools

This will give the necessary to install the pymongo code to support the connection to the database with the python programming language.

sudo easy_install pymongo

I did receive some error message when executing this command and will continue on the programming example from the above referenced website and hopefully will resolve the error, but based on my time constraint – I am pushing on.
.*************************************************************
WARNING: The pymongo._cbson extension module could not
be compiled. No C extensions are essential for PyMongo to run,
although they do result in significant speed improvements.
Above is the ouput showing how the compilation failed.
**************************************************************
At this point of time – I decided to conduct a google search to find the best text environment for the python language or one that will highlight the Python syntax. I usually use Eclipse and probably should start their and see if they support Python.

I did do a little searching and stumbled upon and aptana Pydev extension for eclipse for Python,

http://pydev.org/download.html

So I will try and get this working before I start the coding process of uploading my books.

It is very simple on ubuntu to install eclipse just use

sudo apt-get install eclipse

It was fairly simple to add the extension on Eclipse – I just used the Quick Install method on the pydev.org

Quick Install': Update Manager

Go to the update manager (Help - Install New Software), add:
http://pydev.org/updates

Select PyDev and clicked next and then accepted the license agreement and then I was ready to used Eclipse and Python to start the coding process.
Once you open eclipse and go to New Project and select Pydev Project from the wizard – it will ask you to define an interpreters and I selected among the choices – Iron python, Jython and Python

I gave it a project name MongoDB and used the default home location.
Project type – I selected Python and the grammer version was default 2.6 since I have no preference for grammer versions at this point.

I clicked on configure interpreter and auto config button and it configured by System libs automatically saving me some time.

This is were the confusion started at this point – Eclipse is not as easy to configure as one might think you have to read some documentation to get the environment working so I followed the following information to get this to work -

http://pydev.org/manual_101_root.html

I followed the manual and created a project

file - new - project - Pydev - Pydev project

Project Name: TestMongoDB

Use Defaults

Project Type = Python

GrammerVersion = 2.6

Interpreter Default

Create default 'src' folder and add it to the pythonpath? Is checked

Finish

I then followed the manual to create a python package

File - new - pydev package

I changed the Source Folder to my /TestMongoDB/src and Name = testmongo.test → Finish

After this you will need to create a pydev module

File - new - pydev module

Ensure the source folder is correct

Package name is the same as the one you just created

Template =

Name of the actual script – I called this cbentries
Once my cbentries.py file was created – used the following code to test the db, environment and code
'''
Created on May 1, 2010
@author: Tim
'''
from pymongo.connection import Connection

connection = Connection('localhost')

db = connection.mystorage

doc1 = {"timestamp":001, "msg":"Hello 1"}

doc2 = {"timestamp":002, "msg":"Hello 2"}

doc3 = {"timestamp":003, "msg":"Hello 3"}

db.mystorage.save(doc1)

db.mystorage.save(doc2)

db.mystorage.save(doc3)

cursor = db.mystorage.find()
for d in cursor:
print d

Once the code was modified for mystorage I then clicked on the Run As button and selected

Python Run → Ok

This was the final result of my test – Success – now I can work on getting documents and a user friendly interfact – once this has been completed then I will post the code and probably some lessons learned.

{u'msg': u'Hello 1', u'timestamp': 1, u'_id': ObjectId('4bdcbaee50f9092884000000')}
{u'msg': u'Hello 2', u'timestamp': 2, u'_id': ObjectId('4bdcbaee50f9092884000001')}
{u'msg': u'Hello 3', u'timestamp': 3, u'_id': ObjectId('4bdcbaee50f9092884000002')}

GridFS -----------------------------

I will need to use GridFS to store the acutal files into Mongo.
The database supports native storage of binary data within BSON objects. However, BSON objects in MongoDB are limited to 4MB in size.

The GridFS spec provides a mechanism for transparently dividing a large file among multiple documents. This allows us to efficiently store large objects, and in the case of especially large files, such as videos, permits range operations (e.g., fetching only the first N bytes of a file).

So now the adventure begins in trying to use GridFS to upload my large pdf files,
Reference:

http://dirolf.com/2010/03/29/new-gridfs-implementation-for-pymongo.html

Here is what I have so far using GridFS – this is working great – I will now start the process of searching through a folder and capturing names and extensions of files and auto importing them into my db.
'''

Created on May 1, 2010
@author: Tim
'''
from pymongo import Connection
connection = Connection('localhost')
from gridfs import GridFS
db = connection.mystorage
fs = GridFS(db)
with open("/home/t/Pictures/Photos/warty-final-ubuntu.png") as myimage:
oid = fs.put(myimage, content_type="image/pnp", filename="myimage")
print fs.get(oid)._file #THis will give me the file entries
print fs.get(oid)._id #This will give me the id of the entries
print fs.get(oid).__sizeof__() # This will give me the size of the file in memory
print fs.get(oid).name # This is the name of my file

Enabling a SSH Host and managing Iptable the Ubuntu semi-GUI way

2010-06-12T14:51:00.000-07:00

Iptables is installed by default on Ubuntu 10.04 - this is a new ubuntu build - so last night when I tried to SSH into my new box - I received an port 22: Connection refused -

When I remembered that I do not have an SSH Server and default iptables configured - I needed to complete a couple steps prior to connecting to my server from my laptop.

Step 1.

Install a SSH Server - This is pretty simple using apt-get

sudo apt-get install openssh-server

After all unpacking and RSA key generation is completed - I moved on to the next step of the process .

Step 2

I am pretty lazy and usually do not remember all of the commands to configure my Iptables through the command line, I decided to use firestarter -

sudo apt-get install firestarter

It is probably good to remember how to configure manually - so here it is.

sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT

sudo iptables-save