CISSP $96,000 |
PayScale
Sunday, June 13, 2010
Converts video files to mp4
Here is a quick and dirty post on converting video files to mp4 using a perl script that I wrote and the mp4size ruby script written by Thomer M. Gil
---
------------------------------------------------------------------------------------------------------------------------
#! /usr/bin/perl -wl
# This script will take multiple files in a directory and automatically convert to #.mp4 using .mp4size script -
# Ensure that this ruby and perl script is in the same directory were the files #need to reside
# Written by Tim E
# twitter @ubuntumongol
use strict;
use warnings;
my$file;
my $dir = "/PathtoFiles/";
my $newdir ;
opendir(BIN, $dir) or die "Can't open $dir: $!";
my $newfile;
my $origfile;
while( defined ($file = readdir BIN) ) {
chomp ($file);
my $filex;
my @parts = split (/\./,$file);
$filex = $parts[1];
pop @parts;
my $file_no_ext = join '.', @parts;
$newfile = "\"".$dir.$file_no_ext . ".mp4"."\"";
$origfile = "\"".$dir.$file."\"";
$newdir = "\"".$dir."mp4ize"."\"";
#my $runcommand = "ffmpeg -i $origfile -target ntsc-vcd $newfile";
my $runcommand = "$newdir $origfile";
print "$runcommand\n";
if ($filex eq "mpg")
{
system($runcommand);
}
}
closedir(BIN);
---------------------------------------------------------mp4ize-------------------------------------------------------------------
#!/usr/bin/ruby -w
#
# Copyright (C) 2007-2009 Thomer M. Gil [http://thomer.com/]
#
# Thanks to Brian Moore, Justin Payne, Matt Spitz, Martyn Parker,
# Jean-Francois Macaud, Thomas Hannigan, Anisse Astier, Juanma Hernández,
# Trung Huynh, and Mark Ryan for bugfixes and suggestions.
#
# Oct. 14, 2008: show percentage progress. add -t and -w flags.
# Jan. 11, 2009: switch to bit/s bitrates for newer ffmpeg versions.
# add --iphone option.
# add -y option to ffmpeg (overwrite).
# Jan. 20, 2009: don't exit early when processing multiple files.
# Feb. 17, 2009: deal with "Invalid pixel aspect ratio" error.
# Apr. 1, 2009: new --outdir parameter.
# May 22, 2009: handle filenames with quotes and whitespace.
# Oct 6, 2009: fix bug where we forget to read stderr
# Nov. 5, 2009: fix -v, -t, and -w command line options
# removed bogus 'here' debug statement
#
# This program is free software. You may distribute it under the terms of
# the GNU General Public License as published by the Free Software
# Foundation, version 2.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# This program converts video files to mp4, suitable to be played on an iPod
# or an iPhone. It is careful about maintaining the proper aspect ratio.
#
require 'getoptlong'
require 'open3'
# will automatically try with -vcoded libxvid, also.
# will automatically try with -acodec libfaac, also.
DEFAULT_ARGS = "-f mp4 -y -vcodec xvid -maxrate 1000 -qmin 3 -qmax 5 -g 300 -acodec aac"
DEFAULT_BUFSIZE = 4096
DEFAULT_AUDIO_BITRATE = 128 # will be automatically multiplied with 1024 for newer ffmpeg versions
DEFAULT_VIDEO_BITRATE = 400 # will be automatically multiplied with 1024 for newer ffmpeg versions
IPOD_WIDTH = 320.0
IPOD_HEIGHT = 240.0
IPHONE_WIDTH = 480.0
IPHONE_HEIGHT = 320.0
$options = {}
opts = GetoptLong.new(*[
[ "--audio", "-a", GetoptLong::REQUIRED_ARGUMENT ], # audio bitrate
[ "--help", "-h", GetoptLong::NO_ARGUMENT ], # help
[ "--video", "-b", GetoptLong::REQUIRED_ARGUMENT ], # video bitrate
[ "--verbose", "-v", GetoptLong::NO_ARGUMENT ], # verbose
[ "--width", "-w", GetoptLong::REQUIRED_ARGUMENT ], # override width
[ "--height", "-t", GetoptLong::REQUIRED_ARGUMENT ], # override height
[ "--iphone", "-i", GetoptLong::NO_ARGUMENT ], # set width/height
[ "--outdir", "-o", GetoptLong::REQUIRED_ARGUMENT ], # dir where to write files
])
opts.each { |opt, arg| $options[opt] = arg }
if $options['--help']
puts <
mp4ize - encode videos to mp4 for an iPod or an iPhone
Usage: mp4ize file1.avi [file2.mpg [file3.asf [...]]]
Options:
-h/--help : this help
-v/--verbose : verbose
-a/--audio RATE : override default audio bitrate (#{DEFAULT_AUDIO_BITRATE})
-b/--video RATE : override default video bitrate (#{DEFAULT_VIDEO_BITRATE})
-w/--width WIDTH : over default width (#{IPOD_WIDTH.to_i})
-t/--height HEIGHT : over default height (#{IPOD_HEIGHT.to_i})
-i/--iphone : same as --width #{IPHONE_WIDTH.to_i} --height #{IPHONE_HEIGHT.to_i}
-o/--outdir O : write files to given directory
EOF
exit
end
# --iphone sets --width and --height
if $options['--iphone']
if $options['--width'] || $options['--height']
warn "You can't use --iphone with --width or --height."
exit 1
else
$options['--width'] = $options['-w'] = IPHONE_WIDTH
$options['--height'] = $options['-t'] = IPHONE_HEIGHT
end
end
audio_bitrate = $options['--audio'] || DEFAULT_AUDIO_BITRATE
video_bitrate = $options['--video'] || DEFAULT_VIDEO_BITRATE
ARGV.each do |infile|
outfile = infile.dup
ext = File.extname(outfile)
outfile.sub!(/#{ext}$/, '.mp4')
if $options['--outdir']
if !File.directory?($options['--outdir'])
warn "#{$options['--outdir']} does not exist or is not a directory. exiting."
exit 1
end
outfile = File.join($options['--outdir'], File.basename(outfile))
end
# open the file to figure out the aspect ratio
duration, w, h = 0.0, nil, nil
Open3.popen3("/usr/bin/ffmpeg", "-i", infile) do |stdin, stdout, stderr|
[stdout, stderr].each do |io|
io.each_line do |line|
if line.match(/Video:.+ (\d+)x(\d+)/)
w, h = $1.to_f, $2.to_f
elsif line.match(/Duration:\s+(\d+):(\d+):(\d+)\.(\d+)/)
duration += $1.to_f * 3600
duration += $2.to_f * 60
duration += $3.to_f
duration += $4.to_f / 10
end
end
end
end
begin
aspect = w/h
rescue
puts "Couldn't figure out aspect ratio."
exit
end
user_width = $options['--width'] ? $options['--width'].to_i : IPOD_WIDTH
user_height = $options['--height'] ? $options['--height'].to_i : IPOD_HEIGHT
width = user_width.to_i
height = (width / aspect.to_f).to_i
height -= (height % 2)
pad = ((user_height - height.to_f) / 2.0).to_i
pad -= (pad % 2)
padarg1, padarg2 = "padtop", "padbottom"
# recalculate using the height as the baseline rather than the width
if pad < 0
height = user_height.to_i
width = (height * aspect.to_f).to_i
width -= (width % 2)
pad = ((user_width - width.to_f)/2.0).to_i
pad -= (pad % 2)
padarg1, padarg2 = "padleft", "padright"
end
File.unlink(outfile) if File.exists?(outfile)
# use %infile% and %outfile% and replace those after the split() so that we
# don't split() a filename that has spaces in it.
cmd = "/usr/bin/ffmpeg -i %infile% #{DEFAULT_ARGS} -bufsize #{DEFAULT_BUFSIZE} -s #{width}x#{height} -#{padarg1} #{pad} -#{padarg2} #{pad} -ab #{audio_bitrate} -b #{video_bitrate} %outfile%"
puts cmd if $options['--verbose']
# We could just call "system cmd" here, but we want the exit code of mp4ize
# to tell us whether the duration of the generated mp4 equals the duration
# of the original movie. Exits with a non-zero code if the two are not
# within 1% of each other.
time = 0
STDOUT.sync = true
# try with -vcodec libxvid and -vcodec xvid
# try with -acodec libfaac and -acodec aac
catch(:done) do
5.times do
catch(:retry) do
puts "cmdline: #{cmd}" if $options['--verbose']
cmd_array = cmd.split(/\s+/)
cmd_array.collect! {|s| s.sub(/^%infile%$/, infile)}
cmd_array.collect! {|s| s.sub(/^%outfile%$/, outfile)}
Open3.popen3(*cmd_array) do |stdin, stdout, stderr|
io = select([stdout, stderr], nil, nil, 10)
2.times do |std| # both stdout and stderr
next if io[0][std].nil?
io[0][std].each_line("\r") do |line|
puts "the line is #{line}"
printf("\r%.2f%% | ", time / duration * 100.0)
print line
if line.match(/Invalid pixel aspect ratio/)
cmd.sub!("-s #{width}x#{height}", "-s #{width}x#{height} -aspect #{aspect}")
throw :retry
elsif line.match(/Unknown.*code.*xvid/)
cmd.sub!('-vcodec xvid', '-vcodec libxvid')
throw :retry
elsif line.match(/Unknown.*code.*aac/)
cmd.sub!('-acodec aac', '-acodec libfaac')
throw :retry
# newer ffmpeg versions want bit/s, not kbit/s.
elsif line.match(/The bitrate parameter is set too low/)
cmd.sub!(/-ab \d+/, "-ab #{audio_bitrate}k")
cmd.sub!(/-b \d+/, "-b #{video_bitrate}k")
cmd.sub!(/-bufsize \d+/, "-bufsize #{DEFAULT_BUFSIZE}k")
throw :retry
elsif line.match(/time=([^\s]+)/)
time = $1.to_f
end
end
end
end
throw :done
end
end
end
# return completeness of mp4 file
puts "expected duration: #{duration}" if $options['--verbose']
puts "encoded duration: #{time}" if $options['--verbose']
if ARGV.size == 1
exit((time <= duration * 1.01) && (time >= duration * 0.99))
end
end
---
------------------------------------------------------------------------------------------------------------------------
#! /usr/bin/perl -wl
# This script will take multiple files in a directory and automatically convert to #.mp4 using .mp4size script -
# Ensure that this ruby and perl script is in the same directory were the files #need to reside
# Written by Tim E
# twitter @ubuntumongol
use strict;
use warnings;
my$file;
my $dir = "/PathtoFiles/";
my $newdir ;
opendir(BIN, $dir) or die "Can't open $dir: $!";
my $newfile;
my $origfile;
while( defined ($file = readdir BIN) ) {
chomp ($file);
my $filex;
my @parts = split (/\./,$file);
$filex = $parts[1];
pop @parts;
my $file_no_ext = join '.', @parts;
$newfile = "\"".$dir.$file_no_ext . ".mp4"."\"";
$origfile = "\"".$dir.$file."\"";
$newdir = "\"".$dir."mp4ize"."\"";
#my $runcommand = "ffmpeg -i $origfile -target ntsc-vcd $newfile";
my $runcommand = "$newdir $origfile";
print "$runcommand\n";
if ($filex eq "mpg")
{
system($runcommand);
}
}
closedir(BIN);
---------------------------------------------------------mp4ize-------------------------------------------------------------------
#!/usr/bin/ruby -w
#
# Copyright (C) 2007-2009 Thomer M. Gil [http://thomer.com/]
#
# Thanks to Brian Moore, Justin Payne, Matt Spitz, Martyn Parker,
# Jean-Francois Macaud, Thomas Hannigan, Anisse Astier, Juanma Hernández,
# Trung Huynh, and Mark Ryan for bugfixes and suggestions.
#
# Oct. 14, 2008: show percentage progress. add -t and -w flags.
# Jan. 11, 2009: switch to bit/s bitrates for newer ffmpeg versions.
# add --iphone option.
# add -y option to ffmpeg (overwrite).
# Jan. 20, 2009: don't exit early when processing multiple files.
# Feb. 17, 2009: deal with "Invalid pixel aspect ratio" error.
# Apr. 1, 2009: new --outdir parameter.
# May 22, 2009: handle filenames with quotes and whitespace.
# Oct 6, 2009: fix bug where we forget to read stderr
# Nov. 5, 2009: fix -v, -t, and -w command line options
# removed bogus 'here' debug statement
#
# This program is free software. You may distribute it under the terms of
# the GNU General Public License as published by the Free Software
# Foundation, version 2.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
# Public License for more details.
#
# This program converts video files to mp4, suitable to be played on an iPod
# or an iPhone. It is careful about maintaining the proper aspect ratio.
#
require 'getoptlong'
require 'open3'
# will automatically try with -vcoded libxvid, also.
# will automatically try with -acodec libfaac, also.
DEFAULT_ARGS = "-f mp4 -y -vcodec xvid -maxrate 1000 -qmin 3 -qmax 5 -g 300 -acodec aac"
DEFAULT_BUFSIZE = 4096
DEFAULT_AUDIO_BITRATE = 128 # will be automatically multiplied with 1024 for newer ffmpeg versions
DEFAULT_VIDEO_BITRATE = 400 # will be automatically multiplied with 1024 for newer ffmpeg versions
IPOD_WIDTH = 320.0
IPOD_HEIGHT = 240.0
IPHONE_WIDTH = 480.0
IPHONE_HEIGHT = 320.0
$options = {}
opts = GetoptLong.new(*[
[ "--audio", "-a", GetoptLong::REQUIRED_ARGUMENT ], # audio bitrate
[ "--help", "-h", GetoptLong::NO_ARGUMENT ], # help
[ "--video", "-b", GetoptLong::REQUIRED_ARGUMENT ], # video bitrate
[ "--verbose", "-v", GetoptLong::NO_ARGUMENT ], # verbose
[ "--width", "-w", GetoptLong::REQUIRED_ARGUMENT ], # override width
[ "--height", "-t", GetoptLong::REQUIRED_ARGUMENT ], # override height
[ "--iphone", "-i", GetoptLong::NO_ARGUMENT ], # set width/height
[ "--outdir", "-o", GetoptLong::REQUIRED_ARGUMENT ], # dir where to write files
])
opts.each { |opt, arg| $options[opt] = arg }
if $options['--help']
puts <
mp4ize - encode videos to mp4 for an iPod or an iPhone
Usage: mp4ize file1.avi [file2.mpg [file3.asf [...]]]
Options:
-h/--help : this help
-v/--verbose : verbose
-a/--audio RATE : override default audio bitrate (#{DEFAULT_AUDIO_BITRATE})
-b/--video RATE : override default video bitrate (#{DEFAULT_VIDEO_BITRATE})
-w/--width WIDTH : over default width (#{IPOD_WIDTH.to_i})
-t/--height HEIGHT : over default height (#{IPOD_HEIGHT.to_i})
-i/--iphone : same as --width #{IPHONE_WIDTH.to_i} --height #{IPHONE_HEIGHT.to_i}
-o/--outdir O : write files to given directory
EOF
exit
end
# --iphone sets --width and --height
if $options['--iphone']
if $options['--width'] || $options['--height']
warn "You can't use --iphone with --width or --height."
exit 1
else
$options['--width'] = $options['-w'] = IPHONE_WIDTH
$options['--height'] = $options['-t'] = IPHONE_HEIGHT
end
end
audio_bitrate = $options['--audio'] || DEFAULT_AUDIO_BITRATE
video_bitrate = $options['--video'] || DEFAULT_VIDEO_BITRATE
ARGV.each do |infile|
outfile = infile.dup
ext = File.extname(outfile)
outfile.sub!(/#{ext}$/, '.mp4')
if $options['--outdir']
if !File.directory?($options['--outdir'])
warn "#{$options['--outdir']} does not exist or is not a directory. exiting."
exit 1
end
outfile = File.join($options['--outdir'], File.basename(outfile))
end
# open the file to figure out the aspect ratio
duration, w, h = 0.0, nil, nil
Open3.popen3("/usr/bin/ffmpeg", "-i", infile) do |stdin, stdout, stderr|
[stdout, stderr].each do |io|
io.each_line do |line|
if line.match(/Video:.+ (\d+)x(\d+)/)
w, h = $1.to_f, $2.to_f
elsif line.match(/Duration:\s+(\d+):(\d+):(\d+)\.(\d+)/)
duration += $1.to_f * 3600
duration += $2.to_f * 60
duration += $3.to_f
duration += $4.to_f / 10
end
end
end
end
begin
aspect = w/h
rescue
puts "Couldn't figure out aspect ratio."
exit
end
user_width = $options['--width'] ? $options['--width'].to_i : IPOD_WIDTH
user_height = $options['--height'] ? $options['--height'].to_i : IPOD_HEIGHT
width = user_width.to_i
height = (width / aspect.to_f).to_i
height -= (height % 2)
pad = ((user_height - height.to_f) / 2.0).to_i
pad -= (pad % 2)
padarg1, padarg2 = "padtop", "padbottom"
# recalculate using the height as the baseline rather than the width
if pad < 0
height = user_height.to_i
width = (height * aspect.to_f).to_i
width -= (width % 2)
pad = ((user_width - width.to_f)/2.0).to_i
pad -= (pad % 2)
padarg1, padarg2 = "padleft", "padright"
end
File.unlink(outfile) if File.exists?(outfile)
# use %infile% and %outfile% and replace those after the split() so that we
# don't split() a filename that has spaces in it.
cmd = "/usr/bin/ffmpeg -i %infile% #{DEFAULT_ARGS} -bufsize #{DEFAULT_BUFSIZE} -s #{width}x#{height} -#{padarg1} #{pad} -#{padarg2} #{pad} -ab #{audio_bitrate} -b #{video_bitrate} %outfile%"
puts cmd if $options['--verbose']
# We could just call "system cmd" here, but we want the exit code of mp4ize
# to tell us whether the duration of the generated mp4 equals the duration
# of the original movie. Exits with a non-zero code if the two are not
# within 1% of each other.
time = 0
STDOUT.sync = true
# try with -vcodec libxvid and -vcodec xvid
# try with -acodec libfaac and -acodec aac
catch(:done) do
5.times do
catch(:retry) do
puts "cmdline: #{cmd}" if $options['--verbose']
cmd_array = cmd.split(/\s+/)
cmd_array.collect! {|s| s.sub(/^%infile%$/, infile)}
cmd_array.collect! {|s| s.sub(/^%outfile%$/, outfile)}
Open3.popen3(*cmd_array) do |stdin, stdout, stderr|
io = select([stdout, stderr], nil, nil, 10)
2.times do |std| # both stdout and stderr
next if io[0][std].nil?
io[0][std].each_line("\r") do |line|
puts "the line is #{line}"
printf("\r%.2f%% | ", time / duration * 100.0)
print line
if line.match(/Invalid pixel aspect ratio/)
cmd.sub!("-s #{width}x#{height}", "-s #{width}x#{height} -aspect #{aspect}")
throw :retry
elsif line.match(/Unknown.*code.*xvid/)
cmd.sub!('-vcodec xvid', '-vcodec libxvid')
throw :retry
elsif line.match(/Unknown.*code.*aac/)
cmd.sub!('-acodec aac', '-acodec libfaac')
throw :retry
# newer ffmpeg versions want bit/s, not kbit/s.
elsif line.match(/The bitrate parameter is set too low/)
cmd.sub!(/-ab \d+/, "-ab #{audio_bitrate}k")
cmd.sub!(/-b \d+/, "-b #{video_bitrate}k")
cmd.sub!(/-bufsize \d+/, "-bufsize #{DEFAULT_BUFSIZE}k")
throw :retry
elsif line.match(/time=([^\s]+)/)
time = $1.to_f
end
end
end
end
throw :done
end
end
end
# return completeness of mp4 file
puts "expected duration: #{duration}" if $options['--verbose']
puts "encoded duration: #{time}" if $options['--verbose']
if ARGV.size == 1
exit((time <= duration * 1.01) && (time >= duration * 0.99))
end
end
Saturday, June 12, 2010
Installing MongoDb and Python with eclipse on Ubuntu 9.10
Installing MongoDb and Python with eclipse on Ubuntu 9.10
System = Ubuntu 9.10
Starting Reference = http://giantflyingsaucer.com/blog/?p=839
I just wanted to document a informative and procedural process in setting up python and mongodb and using the eclipse programming environment, it will also be used to reference the python programming project to organize my ebooks and documentation.
I normally will use MySQL for any projects that require a database and have been using MySQL along with the other commercial databases for some years now. But after reading the Linux Journal and learning that MongoDB is a NoSQL database, this has intrigued my interest and since normal databases are not particularly in my opinion suited to store documents – this has led me to start this personal project.
MongoDB Setup Steps
1. Install the curl application to pull the latest db version from the website.
$ apt-get install curl
2. Once curl is installed on the system – create a folder on your local system to serve and the database repository.
$ mkdir /home/t/data/db
3. Once the directory and created then download the latest build from mongodb
$ curl -O http://downloads.mongodb.org/linux/mongodb-linux-i686-latest.tgz
4. Then untar the download from the website
$ tar xzf mongodb-linux-i386-latest.tgz
5. The files will now reside in the /data/db path to start using mongo db simply start the data bast engine by
$ /data/db/mongodb-linux-i686-2010-05-10/bin/mongod &
6. Open another command terminal and execute the following command to get the mongo db interface
$ /data/db/mongodb-linux-i686-2010-05-01/bin/mongo
You will then get an interesting display of text using this version of mongo – so I guess this is the wrong db but the size it good for now
Sat May 1 13:02:23 Mongo DB : starting : pid = 2597 port = 27017 dbpath = /data/db/ master = 0 slave = 0 32-bit
****
WARNING: This is development version of MongoDB. Not recommended for production.
****
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
** see http://blog.mongodb.org/post/137788967/32-bit-limitations for more
At the command prompt of > will be given and from a security perspective unlike MySQL no password is required and I am unsure of the permission the default user is authorized. I will investigate this further after I get through the initial process
The help feature is a strong point for the mongodb – its great to get commands from the system without having to look them up.
7.>help
The help command returned the following results
show dbs show database names
show collections show collections in current database
show users show users in current database
show profile show most recent system.profile entries with time >= 1ms
db.help() help on DB methods
db.foo.help() help on collection methods
db.foo.find() list objects in collection foo
db.foo.find( { a : 1 } ) list objects in foo where a == 1
I used the web reference example about to create a test db entry and return the results
db.mystorage.save( { "message":"Hello World"} )
db.mystorage.findOne()
{ "_id" : ObjectId("4bdc88f8739846d101246a42"), "message" : "Hello World" }
I also added a little more testing and entered two more entries to mystorage database –
db.mystorage.save( { "message1":"Hello World"} )
db.mystorage.save( { "message2":"Hello World"} )
At this point I used the previously mentioned help function and entered the following command to see my entries within the mystorage db that is apparently created by default,
db.mystorage.find()
{ "_id" : ObjectId("4bdc88f8739846d101246a42"), "message" : "Hello World" }
{ "_id" : ObjectId("4bdc8a1c739846d101246a43"), "message1" : "Hello World" }
{ "_id" : ObjectId("4bdc8a21739846d101246a44"), "message2" : "Hello World" }
------------------------------------------------------------------------------------------------
This was part one of my process and now that I know the db portion is working on home development system it is now time to install python and start the programming process of accessing this database and writing an interface for storing and retrieving documents.
I have utilized C, C++, PHP, and Perl as my primary languages of choice and I am most recently refreshed with Perl, but after some reading on the Internet and some language comparison. I decided to give Python a chance for a couple of reasons.
1. Easier to read by a human compared to the cryptic nature of Perl.
2. I am not really creating a web programming structure so PHP is out the question.
3. I am not concerned with speed for this project and even though C++ is probably my favorite language it would be an overkill for this simple adventure.
So I have a new language to learn that has been mentioned to be more secure and easy for someone with Perl and C++ programming experience.
I will try and define some differences and provide an overall transition assessment of using a language that I have never used before in my previous experiences.
------------------------------------------------------------------------------------------------
Installing Python on my Ubuntu system.
The will start by installing the python-setuptools using the following commands
sudo apt-get install python-setuptools
This will give the necessary to install the pymongo code to support the connection to the database with the python programming language.
sudo easy_install pymongo
I did receive some error message when executing this command and will continue on the programming example from the above referenced website and hopefully will resolve the error, but based on my time constraint – I am pushing on.
.*************************************************************
WARNING: The pymongo._cbson extension module could not
be compiled. No C extensions are essential for PyMongo to run,
although they do result in significant speed improvements.
Above is the ouput showing how the compilation failed.
**************************************************************
At this point of time – I decided to conduct a google search to find the best text environment for the python language or one that will highlight the Python syntax. I usually use Eclipse and probably should start their and see if they support Python.
I did do a little searching and stumbled upon and aptana Pydev extension for eclipse for Python,
http://pydev.org/download.html
So I will try and get this working before I start the coding process of uploading my books.
It is very simple on ubuntu to install eclipse just use
sudo apt-get install eclipse
It was fairly simple to add the extension on Eclipse – I just used the Quick Install method on the pydev.org
Quick Install': Update Manager
Go to the update manager (Help - Install New Software), add:
http://pydev.org/updates
Select PyDev and clicked next and then accepted the license agreement and then I was ready to used Eclipse and Python to start the coding process.
Once you open eclipse and go to New Project and select Pydev Project from the wizard – it will ask you to define an interpreters and I selected among the choices – Iron python, Jython and Python
I gave it a project name MongoDB and used the default home location.
Project type – I selected Python and the grammer version was default 2.6 since I have no preference for grammer versions at this point.
I clicked on configure interpreter and auto config button and it configured by System libs automatically saving me some time.
This is were the confusion started at this point – Eclipse is not as easy to configure as one might think you have to read some documentation to get the environment working so I followed the following information to get this to work -
http://pydev.org/manual_101_root.html
I followed the manual and created a project
file - new - project - Pydev - Pydev project
Project Name: TestMongoDB
Use Defaults
Project Type = Python
GrammerVersion = 2.6
Interpreter Default
Create default 'src' folder and add it to the pythonpath? Is checked
Finish
I then followed the manual to create a python package
File - new - pydev package
I changed the Source Folder to my /TestMongoDB/src and Name = testmongo.test → Finish
After this you will need to create a pydev module
Once my cbentries.py file was created – used the following code to test the db, environment and code
'''
Created on May 1, 2010
@author: Tim
'''
from pymongo.connection import Connection
connection = Connection('localhost')
db = connection.mystorage
doc1 = {"timestamp":001, "msg":"Hello 1"}
doc2 = {"timestamp":002, "msg":"Hello 2"}
doc3 = {"timestamp":003, "msg":"Hello 3"}
db.mystorage.save(doc1)
db.mystorage.save(doc2)
db.mystorage.save(doc3)
cursor = db.mystorage.find()
for d in cursor:
print d
Once the code was modified for mystorage I then clicked on the Run As button and selected
Python Run → Ok
This was the final result of my test – Success – now I can work on getting documents and a user friendly interfact – once this has been completed then I will post the code and probably some lessons learned.
{u'msg': u'Hello 1', u'timestamp': 1, u'_id': ObjectId('4bdcbaee50f9092884000000')}
{u'msg': u'Hello 2', u'timestamp': 2, u'_id': ObjectId('4bdcbaee50f9092884000001')}
{u'msg': u'Hello 3', u'timestamp': 3, u'_id': ObjectId('4bdcbaee50f9092884000002')}
GridFS -----------------------------
I will need to use GridFS to store the acutal files into Mongo.
The database supports native storage of binary data within BSON objects. However, BSON objects in MongoDB are limited to 4MB in size.
The GridFS spec provides a mechanism for transparently dividing a large file among multiple documents. This allows us to efficiently store large objects, and in the case of especially large files, such as videos, permits range operations (e.g., fetching only the first N bytes of a file).
So now the adventure begins in trying to use GridFS to upload my large pdf files,
Reference:
http://dirolf.com/2010/03/29/new-gridfs-implementation-for-pymongo.html
Here is what I have so far using GridFS – this is working great – I will now start the process of searching through a folder and capturing names and extensions of files and auto importing them into my db.
'''
Created on May 1, 2010
@author: Tim
'''
from pymongo import Connection
connection = Connection('localhost')
from gridfs import GridFS
db = connection.mystorage
fs = GridFS(db)
with open("/home/t/Pictures/Photos/warty-final-ubuntu.png") as myimage:
oid = fs.put(myimage, content_type="image/pnp", filename="myimage")
print fs.get(oid)._file #THis will give me the file entries
print fs.get(oid)._id #This will give me the id of the entries
print fs.get(oid).__sizeof__() # This will give me the size of the file in memory
print fs.get(oid).name # This is the name of my file
System = Ubuntu 9.10
Starting Reference = http://giantflyingsaucer.com/blog/?p=839
I just wanted to document a informative and procedural process in setting up python and mongodb and using the eclipse programming environment, it will also be used to reference the python programming project to organize my ebooks and documentation.
I normally will use MySQL for any projects that require a database and have been using MySQL along with the other commercial databases for some years now. But after reading the Linux Journal and learning that MongoDB is a NoSQL database, this has intrigued my interest and since normal databases are not particularly in my opinion suited to store documents – this has led me to start this personal project.
MongoDB Setup Steps
1. Install the curl application to pull the latest db version from the website.
$ apt-get install curl
2. Once curl is installed on the system – create a folder on your local system to serve and the database repository.
$ mkdir /home/t/data/db
3. Once the directory and created then download the latest build from mongodb
$ curl -O http://downloads.mongodb.org/linux/mongodb-linux-i686-latest.tgz
4. Then untar the download from the website
$ tar xzf mongodb-linux-i386-latest.tgz
5. The files will now reside in the /data/db path to start using mongo db simply start the data bast engine by
$ /data/db/mongodb-linux-i686-2010-05-10/bin/mongod &
6. Open another command terminal and execute the following command to get the mongo db interface
$ /data/db/mongodb-linux-i686-2010-05-01/bin/mongo
You will then get an interesting display of text using this version of mongo – so I guess this is the wrong db but the size it good for now
Sat May 1 13:02:23 Mongo DB : starting : pid = 2597 port = 27017 dbpath = /data/db/ master = 0 slave = 0 32-bit
****
WARNING: This is development version of MongoDB. Not recommended for production.
****
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
** see http://blog.mongodb.org/post/137788967/32-bit-limitations for more
At the command prompt of > will be given and from a security perspective unlike MySQL no password is required and I am unsure of the permission the default user is authorized. I will investigate this further after I get through the initial process
The help feature is a strong point for the mongodb – its great to get commands from the system without having to look them up.
7.>help
The help command returned the following results
show dbs show database names
show collections show collections in current database
show users show users in current database
show profile show most recent system.profile entries with time >= 1ms
db.help() help on DB methods
db.foo.help() help on collection methods
db.foo.find() list objects in collection foo
db.foo.find( { a : 1 } ) list objects in foo where a == 1
I used the web reference example about to create a test db entry and return the results
db.mystorage.save( { "message":"Hello World"} )
db.mystorage.findOne()
{ "_id" : ObjectId("4bdc88f8739846d101246a42"), "message" : "Hello World" }
I also added a little more testing and entered two more entries to mystorage database –
db.mystorage.save( { "message1":"Hello World"} )
db.mystorage.save( { "message2":"Hello World"} )
At this point I used the previously mentioned help function and entered the following command to see my entries within the mystorage db that is apparently created by default,
db.mystorage.find()
{ "_id" : ObjectId("4bdc88f8739846d101246a42"), "message" : "Hello World" }
{ "_id" : ObjectId("4bdc8a1c739846d101246a43"), "message1" : "Hello World" }
{ "_id" : ObjectId("4bdc8a21739846d101246a44"), "message2" : "Hello World" }
------------------------------------------------------------------------------------------------
This was part one of my process and now that I know the db portion is working on home development system it is now time to install python and start the programming process of accessing this database and writing an interface for storing and retrieving documents.
I have utilized C, C++, PHP, and Perl as my primary languages of choice and I am most recently refreshed with Perl, but after some reading on the Internet and some language comparison. I decided to give Python a chance for a couple of reasons.
1. Easier to read by a human compared to the cryptic nature of Perl.
2. I am not really creating a web programming structure so PHP is out the question.
3. I am not concerned with speed for this project and even though C++ is probably my favorite language it would be an overkill for this simple adventure.
So I have a new language to learn that has been mentioned to be more secure and easy for someone with Perl and C++ programming experience.
I will try and define some differences and provide an overall transition assessment of using a language that I have never used before in my previous experiences.
------------------------------------------------------------------------------------------------
Installing Python on my Ubuntu system.
The will start by installing the python-setuptools using the following commands
sudo apt-get install python-setuptools
This will give the necessary to install the pymongo code to support the connection to the database with the python programming language.
sudo easy_install pymongo
I did receive some error message when executing this command and will continue on the programming example from the above referenced website and hopefully will resolve the error, but based on my time constraint – I am pushing on.
.*************************************************************
WARNING: The pymongo._cbson extension module could not
be compiled. No C extensions are essential for PyMongo to run,
although they do result in significant speed improvements.
Above is the ouput showing how the compilation failed.
**************************************************************
At this point of time – I decided to conduct a google search to find the best text environment for the python language or one that will highlight the Python syntax. I usually use Eclipse and probably should start their and see if they support Python.
I did do a little searching and stumbled upon and aptana Pydev extension for eclipse for Python,
http://pydev.org/download.html
So I will try and get this working before I start the coding process of uploading my books.
It is very simple on ubuntu to install eclipse just use
sudo apt-get install eclipse
It was fairly simple to add the extension on Eclipse – I just used the Quick Install method on the pydev.org
Quick Install': Update Manager
Go to the update manager (Help - Install New Software), add:
http://pydev.org/updates
Select PyDev and clicked next and then accepted the license agreement and then I was ready to used Eclipse and Python to start the coding process.
Once you open eclipse and go to New Project and select Pydev Project from the wizard – it will ask you to define an interpreters and I selected among the choices – Iron python, Jython and Python
I gave it a project name MongoDB and used the default home location.
Project type – I selected Python and the grammer version was default 2.6 since I have no preference for grammer versions at this point.
I clicked on configure interpreter and auto config button and it configured by System libs automatically saving me some time.
This is were the confusion started at this point – Eclipse is not as easy to configure as one might think you have to read some documentation to get the environment working so I followed the following information to get this to work -
http://pydev.org/manual_101_root.html
I followed the manual and created a project
file - new - project - Pydev - Pydev project
Project Name: TestMongoDB
Use Defaults
Project Type = Python
GrammerVersion = 2.6
Interpreter Default
Create default 'src' folder and add it to the pythonpath? Is checked
Finish
I then followed the manual to create a python package
File - new - pydev package
I changed the Source Folder to my /TestMongoDB/src and Name = testmongo.test → Finish
After this you will need to create a pydev module
- File - new - pydev module
- Ensure the source folder is correct
- Package name is the same as the one you just created
- Template =
Once my cbentries.py file was created – used the following code to test the db, environment and code
'''
Created on May 1, 2010
@author: Tim
'''
from pymongo.connection import Connection
connection = Connection('localhost')
db = connection.mystorage
doc1 = {"timestamp":001, "msg":"Hello 1"}
doc2 = {"timestamp":002, "msg":"Hello 2"}
doc3 = {"timestamp":003, "msg":"Hello 3"}
db.mystorage.save(doc1)
db.mystorage.save(doc2)
db.mystorage.save(doc3)
cursor = db.mystorage.find()
for d in cursor:
print d
Once the code was modified for mystorage I then clicked on the Run As button and selected
Python Run → Ok
This was the final result of my test – Success – now I can work on getting documents and a user friendly interfact – once this has been completed then I will post the code and probably some lessons learned.
{u'msg': u'Hello 1', u'timestamp': 1, u'_id': ObjectId('4bdcbaee50f9092884000000')}
{u'msg': u'Hello 2', u'timestamp': 2, u'_id': ObjectId('4bdcbaee50f9092884000001')}
{u'msg': u'Hello 3', u'timestamp': 3, u'_id': ObjectId('4bdcbaee50f9092884000002')}
GridFS -----------------------------
I will need to use GridFS to store the acutal files into Mongo.
The database supports native storage of binary data within BSON objects. However, BSON objects in MongoDB are limited to 4MB in size.
The GridFS spec provides a mechanism for transparently dividing a large file among multiple documents. This allows us to efficiently store large objects, and in the case of especially large files, such as videos, permits range operations (e.g., fetching only the first N bytes of a file).
So now the adventure begins in trying to use GridFS to upload my large pdf files,
Reference:
http://dirolf.com/2010/03/29/new-gridfs-implementation-for-pymongo.html
Here is what I have so far using GridFS – this is working great – I will now start the process of searching through a folder and capturing names and extensions of files and auto importing them into my db.
'''
Created on May 1, 2010
@author: Tim
'''
from pymongo import Connection
connection = Connection('localhost')
from gridfs import GridFS
db = connection.mystorage
fs = GridFS(db)
with open("/home/t/Pictures/Photos/warty-final-ubuntu.png") as myimage:
oid = fs.put(myimage, content_type="image/pnp", filename="myimage")
print fs.get(oid)._file #THis will give me the file entries
print fs.get(oid)._id #This will give me the id of the entries
print fs.get(oid).__sizeof__() # This will give me the size of the file in memory
print fs.get(oid).name # This is the name of my file
Enabling a SSH Host and managing Iptable the Ubuntu semi-GUI way
Iptables is installed by default on Ubuntu 10.04 - this is a new ubuntu build - so last night when I tried to SSH into my new box - I received an port 22: Connection refused -
When I remembered that I do not have an SSH Server and default iptables configured - I needed to complete a couple steps prior to connecting to my server from my laptop.
Step 1.
Install a SSH Server - This is pretty simple using apt-get
sudo apt-get install openssh-server
After all unpacking and RSA key generation is completed - I moved on to the next step of the process .
Step 2
I am pretty lazy and usually do not remember all of the commands to configure my Iptables through the command line, I decided to use firestarter -
sudo apt-get install firestarter
It is probably good to remember how to configure manually - so here it is.
sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT
sudo iptables-save
When I remembered that I do not have an SSH Server and default iptables configured - I needed to complete a couple steps prior to connecting to my server from my laptop.
Step 1.
Install a SSH Server - This is pretty simple using apt-get
sudo apt-get install openssh-server
After all unpacking and RSA key generation is completed - I moved on to the next step of the process .
Step 2
I am pretty lazy and usually do not remember all of the commands to configure my Iptables through the command line, I decided to use firestarter -
sudo apt-get install firestarter
It is probably good to remember how to configure manually - so here it is.
sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT
sudo iptables-save
Subscribe to:
Posts (Atom)