Friday, September 3, 2010

Learning MySQL and PHP

  • PHP and MySQL are two different languages that work hand in hand very well.
  • In order to get us started the right way we will need our database up and running and the web server to be able to run php code. XAMPP is an excellent tool to put your localhost "alive" in under 5 minutes.
  • We will need 3 informations in order to access and manipulate our databases: host ("localhost" usually), username and password (in XAMPP this comes by default with username "root" and no password.
  • At first, we stored our host name (localhost), username and password in variables to call and update them a little bit easier (I can use the variable '$host' in 1000 pages and update it by only modifying this step instead of modifying all those pages one by one, that's why we store the info in variables).
  • Next we define '$connect' with the actual connection code. This is very important because it's a little different from most of the examples out there.
  • Notice the '@' character right at the start of the 'mysql_connect' command. It's used to hide the errors that php produces if any of the variables contain informations that are not correct and the script can't connect to the host.
  • This is a very important step in writing secure code that will not leave empty doors.
  • The errors can output a lot of sensitive informations and we don't wanna put that kind of info in the wrong hands so we will hide the default errors and put our own messages to identify where the error is.
  • Let's take a live example for a better understanding: Instead of 'root' as the database username, I will put 'rosot' and delete the '@' that is supposed to hide my errors. Here's the ugly output: 'Warning: mysql_connect() [function.mysql-connect]: Access denied for user 'rosot'@'localhost' (using password: YES) in D:\xampp\htdocs\test\index.php on line 9' It gives out my username, it's not a big thing since it's the wrong one but keep in mind that the same error will be shown if we enter a wrong password so it's not safe.
  • Let's put back the '@' character and run this code again. The output is somehow the same because I defined this way but you can change it to whatever you want since you're in full control. We managed to control the errors and display only what we need by using an if statement.
  • By default, the command 'mysql_query' executes a query the we define and store in a variable or inside the parenthesis (it's the same thing). This mysql command returns TRUE or FALSE (TRUE = I executed the query successfully, FALSE = no I didin't) and this boolean return will help us figure out if we should display an error message or success.
  • if ( ! @mysql_query ( "CREATE DATABASE `roscripts`" ) )
    Another character (!) is joining the game: if ( ! //code here ) means that if the code that follows the character is not true than the condition is true. In pure english it goes this way: "If I can't execute this query, I do this...". This condition is the same with the following but a little more elegant:
    if ( @mysql_query ( "CREATE DATABASE `roscripts`" ) == FALSE )
    So if the code can't execute the query command, it hides the default errors (thanks to '@') and displays the message that I want (in our case the code dies and displays the mysql error : 'die ( mysql_error() );').

Sample Application that takes a word and Enters into database.

  1. create a empty php file and name it as index.php
  2. lets have a text box to enter the word and 2 buttons, one for submit and other for reset.




  3. My Term Storage



    " method="post">


    First name






  4. In form action we are writing , because we are going to write the php functionality in the same page.
  5. Now we have to write a functionality in php which will insert the word entered in text box to a specified table in the mysql database.
  6. Specifying the database parameters
  7. $database="mystore";
    $conn = @mysql_connect( 'localhost', 'username', 'password' ) or die( mysql_errno().': '.mysql_error().NL );
    @mysql_select_db($database) or die( "Unable to select database");
  8. Now get the value of text box into a variable $name = $_POST['first'];
  9. Build a sql query :$sql = "INSERT INTO terms VALUES ('$name',NULL,NULL,NULL)";
  10. Run the query
  11. $result = mysql_query( $sql,$conn );
    mysql_free_result( $result );
  12. Thus, we have inserted the record successfully.
Display of data from mysql table onto a html page.
  1. Now i want the display of records in the same page as and when i insert.
  2. So, we need to write a code for displaying the data in mysql table onto html page.
  3. For getting the data, we need to fire a sql query "select * from terms".
  4. Build the query :$query="select * from terms";
  5. execute the query and store the result set in a variable.
  6. $rt=mysql_query($query);
  7. putting the data into a unordered list.
  8. echo "
      ";
      while($nt=mysql_fetch_array($rt)){
      echo "
    • $nt[term]
    • "; //nt will contain each row.
      }
      echo "
    ";


Friday, August 27, 2010

Information Extraction versus Information Retrieval

Information extraction (IE). Unlike information retrieval (IR), which concerns how to identify relevant documents from a document collection, IE produces structured data ready for post-processing, which is crucial to many applications of Web mining and searching tools.

Programs that perform the task of IE are referred to as extractors or wrappers. A wrapper was originally defined as a component in an information integration system which aims at providing a single uniform query interface to access multiple information sources. In an information integration system, a wrapper is generally a program that “wraps” an information source (e.g. a database server, or a Web server) such that the information integration system can access that information source without changing its core query answering mechanism.

Wrapper induction (WI) or information extraction (IE) systems are software tools that are designed to generate wrappers.


Source: Survey of Web Information Extraction Systems
Traditional IE aims at extracting data from totally unstructured free texts that are written in natural language. Web IE, in contrast, processes online documents that are semi-structured and usually generated automatically by a server-side application program. As a result, traditional IE usually takes advantage of NLP techniques such as lexicons and grammars, whereas Web IE usually applies machine learning and pattern mining techniques to exploit the syntactical patterns or layout structures of the template-based documents.

There are five main tasks defined for text IE, including named entity recognition, coreference resolution, template element construction, template relation construction and scenario template production.

RISE (Repository of Online Information Sources Used in Information Extraction Tasks).

Classification

Message Understanding Conferences (MUCs) have classified as MUC Approaches and Post MUC Approaches.

MUC Approaches:
  1. AutoSolg
  2. LIEP
  3. PALKA
  4. HASTEN
  5. CRYSTAL
Post MUC Approaches:
  1. WHISK
  2. RAPIER
  3. SRV
  4. WIEN
  5. SoftMealy
  6. STALKER
Hsu and Dung classified into 4 categories, hand-crafted wrappers using general programming languages, specially designed programming languages or tools, heuristic-based wrappers, and WI approaches.

Chang classified based on the degree of automation, They classified IE tools into four distinct categories, including systems that need programmers, systems that need annotation examples, annotation-free systems and semisupervised systems.

Mulsea classified IE tools into 3 classes:
IE Tools based on
  1. Syntactic/Semantic Constraints
  2. Delimiters
  3. Both 1 and 2.
Kushmerick classified many of the IE tools into two distinct categories finite-state and relational learning tools.

Laender proposed a taxonomy for data extraction tools based on the main technique used by each tool to generate a wrapper, which are
  1. Languages for Wrapper Development.
  2. HTML-Aware Tools.
  3. NLP-Based tools.
  4. Wrapper Induction tools.
  5. Modeling based tools.
  6. Ontology based tools.
Laender compared among the tools by using the following 7 features: degree of automation, support for complex objects, page contents, availability of a GUI, XML output, support for non-HTML sources, resilience, and adaptiveness.

Sarawagi classified HTML wrappers into 3 categories according to the kind of extraction tasks.
  1. Record level wrappers. - exploits regularities to discover record boundaries and then extract elements of a single list of homogeneous records from a page.
  2. Page level wrappers. - extracts elements of multiple kinds of records.
  3. Site level wrappers. - populate a database from pages of a Web site.

Information Extraction versus Information Retrieval

Information extraction (IE) is unlike Information Retrieval (IR), which concerns how to identify relevant documents from a document collection, IE produces structured data ready for post-processing, which is crucial to many applications of Web mining and searching tools.

Programs that perform the task of IE are referred to as extractors or wrappers. A wrapper was originally defined as a component in an information integration system which aims at providing a single uniform query interface to access multiple information sources. In an information integration system, a wrapper is generally a program that “wraps” an information source (e.g. a database server, or a Web server) such that the information integration system can access that information source without changing its core query answering mechanism.




Friday, August 13, 2010

Github Configurations

GETTING A PUBLIC KEY

First check to see if a ssh key directory exists.

$ cd ~/.ssh
$ ls
config                id_rsa.pub
id_rsa known_hosts
$ mkdir key_backup
$ cp id_rsa* key_backup
$ rm id_rsa*

Here we have an existing keypair, id_rsa and id_rsa.pub, which we’ve copied into ~/.ssh/key_backup before removing. By default, ssh will use keys in ~/.ssh that are named id_rsa, id_dsa or identity.

Generating a key

If you have an existing keypair you wish to use, you can skip this step.

Now that we’re certain ssh won’t use an existing key, it’s time to generate a new keypair. Lets make an RSA keypair:

$ ssh-keygen -t rsa -C "tekkub@gmail.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tekkub/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tekkub/.ssh/id_rsa.
Your public key has been saved in /home/tekkub/.ssh/id_rsa.pub.
The key fingerprint is:
01:0f:f4:3b:ca:85:d6:17:a1:7d:f0:68:9d:f0:a2:db tekkub@gmail.com

At the first prompt you can just hit enter to generate the key with the default name. You should use a good passphrase with your key. See Working with SSH key passphrases for more details on why you should use a passphrase and how to avoid re-entering it every time you use your key.

Note: If you don’t use the default key names, or store your keys in a different path, you may need to run ssh-add path/to/my_key so that ssh knows where to find your key.

Adding the key to your GitHub account

Now launch your browser and open the account page. In the “SSH Public Keys” section click “add another public key”, then paste your public key into the “key” field. If you leave the title blank the key comment (your email) will be used for the title.

REPOSITORY SETUP:

Global setup:

 Download and install Git
git config --global user.name "Vamshi Krishna Reddy V"
git config --global user.email vamshi4001@gmail.com

Next steps:

 mkdir Twinsight
cd Twinsight
git init
touch README
git add README
git commit -m 'first commit'
git remote add origin git@github.com:vamshi4001/Twinsight.git
git push origin master

Existing Git Repo?

 cd existing_git_repo
git remote add origin git@github.com:vamshi4001/Twinsight.git
git push origin master

Import a Subversion Repository

  • If the repo you are importing is very large, your import may time out.
  • If your subversion repository contains a non-standard directory structure, this import process will probably not work for you.
  • This service currently only supports public subversion repositories.
  • You can find details on how to run a manual import here.

SVN Repository URL help

Thursday, August 12, 2010

Blog Related Stuff

http://blogcosm.com/ - A daily reference to the world of blogs.
Blog Classification NLP[pdf]

Twitter Bots

I love Twitter. Why? Quite simply, because of the amazing community of helpful, knowledgeable, and diverse people around Twitter. I’ve already met many interesting individuals, both on and offline, because we all participate in the Twitter community. And Twitter keeps getting better through innovation from the community. There is a growing trend here, the use of Twitter bots.

What are Twitter Bots? They are special Twitter accounts that perform a special function and provides you with useful information. Twitter bots come in 2 basic flavors:

1) Push Bots - These bots don’t do anything fancy. Once you start to follow them, they broadcast messages to you. The most common uses of push bots to date have been by sports teams (for scoring updates) and weather forecasts.

2) Pull Bots – These bots are more sophisticated than push bots. You can interact with them by using Direct Message commands. The features of pull bots greatly exceed push bots. Pull bots operate as micro-applications behind the scenes, processing commands from a Twitter user, doing some work and then sending the result/data back to the original user via a Direct Message.

A simple example is the Timer bot. It’s Twitters version of a personal reminder service. Here’s how it works:

  • Follow Timer bot
  • Send a direct message to timer like this “d timer 45 call mom’

The timer bot will save your request, wait 45 minutes before sending a direct message reminder back to your Twitter account that says ‘call mom’

Here is a list of some of the more popular Twitter Bots. If you have others you find useful, add them to the comments and I will update the list.

  • gCal - Add Google Calendar events
  • HappyTwitDay - Send Happy Birthday Wishes
  • WineTweets - Share what you’re drinking with other wine lovers on Twitter (check this out @garyvee)
  • TweetBeep - Track topics in Twitter and have them emailed to you
  • Tipr - Tells you the amount you should leave for a tip when dining out
Source: http://www.kenburbary.com/2008/06/twitter-bots-usage-steadily-growing/

Apache Cassandra

Cassandra was open sourced by Facebook in 2008, and is now developed by Apache committers and contributors from many companies.

SOME DEFINITIONS:


Stable releases

Cassandra stable releases are well tested and reasonably free of serious problems, (or at least the problems are known and well documented). If you are setting up a production environment, a stable release is what you want.

Betas and release candidates

Betas are prototype releases considered ready for user testing, and release candidates have the potential to become the next stable release. These releases represent the state-of-the-art so are often the best place to start, and since APIs and on-disk storage formats can change between major versions this can also save you from an upgrade. The testing and feedback is also highly appreciated.

Nightly builds

Nightly builds represent the current state of development as of the time of the build. They contain all of the previous day's new features, fixes, and newly introduced bugs. The only guarantee they come with is that they successfully build and the unit tests pass. Nightly builds are a handy way of testing recent changes, or accessing the latest features and fixes not found in beta or release candidates, but there is some risk of them being buggy.

Tuesday, August 3, 2010

Virus - USB files seen as shortcuts

Virus Problem and Solution

Issue


I caught a virus on my flash drive at work and it appears to have changed all my file names to short cuts. I believe I've cleaned the virus but how do i get my files back so that I can view them?

Solution

  • If you did not format your flash drive, then check whether the files are not in hidden mode.
  • Click on "Start" -->Run-->type cmd and click on OK.
  • Here I assume your pendrive as G:
  • Enter this command.
  • attrib -h -r -s /s /d g:\*.*
  • Note : Replace the letter g with your flash drive letter.


Saturday, July 17, 2010

Links Collection

Term Extraction

Zemanta - term extractor
Steve makes websites
Some Examples of yahoo term extraction
Five Filters term extraction
POS tagger term extraction

Free software for term extraction:

Non-free web services for term extraction:



YUI Slider example

Twitter css n jquery
http://www.dynamicwp.net/articles-and-tutorials/10-awesome-jquery-tutorials-about-twitter/
http://psdcollector.blogspot.com/2010/03/useful-jquery-twitter-api-tutorials.html
http://craigsworks.com/projects/qtip/
http://cufon.shoqolate.com/generate/

http://www.marcofolio.net/
http://archive.usejquery.com/submit.html
http://www.webspeaks.in/2010/07/25-incredibly-useful-css-tricks-you.html
http://www.readwriteweb.com/archives/10_twitter_list_widgets_you_can_grab_embed_right_n.php
http://www.markhneedham.com/blog/2010/03/21/node-js-a-little-application-with-twitter-couchdb/

interesting app

http://www.adictomatic.com

jQuery twitter search plugin

http://jquery.malsup.com/twitter/

What to do if starting webdesign

http://www.webresourcesdepot.com/if-i-had-started-web-design-today-and-you/

New updates


http://critiquethesite.com/
http://mashable.com/2010/07/22/web-design-feedback-tools/
http://www.moretechtips.net/

Nice OAuth Tutorial

http://www.snipe.net/2009/07/writing-your-first-twitter-application-with-oauth/#axzz0ub1EY049


Interesting Ones

http://custom-drupal.com/jquery-demo/
http://www.geekdaily.net/category/projects/
http://visionmedia.github.com/inline-search/
http://www.hiddenpixels.com/designer-and-developer-resources/jquery-examples/
http://www.deontee.com/47-top-twitter-sites-services-software-and-tips/
http://www.learningjquery.com/2010/07/great-ways-to-learn-jquery
http://www.9lessons.info/2010/04/free-smtp-mail-server-for-your-web.html
http://elitebydesign.com/getting-started-with-css-20-awesome-introductory-tuts/
http://sorgalla.com/projects/jcarousel/
http://www.sohtanaka.com/web-design/category/css-tutorials/page/4/
10 Tools for Distributed Developer Teams

Envolve: A Facebook-Like Chat Room for Your Site

http://trends.ellerdale.com/trends/people
http://free-idea-bank.blogspot.com/?expref=next-blog

HPRICOT:

http://wiki.github.com/hpricot/hpricot/hpricot-challenge
http://totalrecall.wordpress.com/2008/08/30/extracting-data-with-hpricot-night-2/
http://www.sitepoint.com/blogs/2007/11/21/processing-html-with-hpricot/
http://stackoverflow.com/questions/1972934/searching-hpricot-with-regex
http://stackoverflow.com/questions/tagged/hpricot
http://stackoverflow.com/questions/667697/parsing-an-html-table-using-hpricot-ruby
http://code.google.com/p/twitcurl/wiki/WikiHowToUseTwitcurlLibrary
http://www.rubyinside.com/fast-html-parsing-in-ruby-with-hpricot-125.html
http://www.freebsdsoftware.org/textproc/ruby-html-parser.html
http://ai-depot.com/articles/the-easy-way-to-extract-useful-text-from-arbitrary-html/


MAIL

http://www.simplehelp.net/2008/12/01/how-to-send-email-from-the-linux-command-line/


NEWS TICKER
http://www.smartmenus.org/vadikom/products/free-scripts/demos/v_ex3.html

INTRESTING ONE's
http://www.engageinteractive.co.uk/

http://www.tiddlywiki.com/ nice one have a look
http://www.netzgesta.de


http://www.linuxjournal.com/article/8110
http://www.linuxjournal.com/article/8130http://kernelnewbies.org/




Thursday, February 18, 2010

Basic SVN Commands

http://www.linuxfromscratch.org/blfs/edguide/chapter03.html#ch03-propset

Find out runlevel of unix or Linux system

A runlevel is a software configuration of the system which allows only a selected group of processes to exist. The processes spawned by init command/process for each of these runlevels are defined in the /etc/inittab file. Runlevels 0, 1, and 6 are reserved. Runlevel 0 is used to halt the sys tem, runlevel 6 is used to reboot the system, and runlevel 1 is used to get the system down into single user mode. In order to print current runlevel you need to use command who or runlevel as follows:

1) Print print current runlevel using who command:

$ who -r

run-level 2  Dec 16 11:45                   last=S
2) Find the current and previous system runlevel using runlevel command:

$ runlevel

N 2 

Runlevel reads the system utmp file (typically /var/run/utmp) to locate the runlevel record, and then prints the previous and current system runlevel on its screen. Runlevel can be used in rc scripts as a substitute for the System-V who -r command. However, in newer versions of init this information is also available in the environment variables RUNLEVEL and PREVLEVEL:

$ echo $RUNLEVEL
$ echo $PREVLEVEL

Source: http://www.cyberciti.biz/howto/question/linux/unix-linux-find-out-runlevel.php

Linux change the runlevel for a server without rebooting

Q. How do I change the runlevel for a Linux server without rebooting my server? I am using Gentoo Linux.

A. Almost all Linux distribution use init program to change runlevel. init is the parent of all processes. Its primary role is to create processes from a script stored in the file
/etc/inittab . This file usually has entries which cause init to spawn gettys on each line that users can log in. It also controls autonomous processes required by any particular system.

To change runlevel from 3 to 1 you need to type command as follows:
# init 1
OR
$ sudo init 1

Where 1 is runlevel number, other possible runlevel values are:
=> 0 - Shutdown server
=> 1 - Take server to single-user mode
=> 2 - Start X Window (used by Debian)
=> 3 - Full multi-user text mode
=> 5 - Start X Window
=> 6 - Reboot server


Source: http://www.cyberciti.biz/faq/change-the-runlevel-for-a-linux-server/

Changing runlevels

More Information on Runlevels: http://en.wikipedia.org/wiki/Runlevel

Wednesday, February 17, 2010

Execute a command without saving it in the history

$command
This is sample output - yours may be different.
$ echo this goes to history
this goes to history

$ echo this wont go to history
this wont go to history

$ history
1 echo this goes to history
2 history

Monday, February 8, 2010

Twittering with UBUNTU

http://jamsubuntu.blogspot.com/2009/03/twittering-with-ubuntu.html

Sunday, February 7, 2010

Running Scripts on Shutdown and Startup

Running Script at time of shutdown
=========================

myscript:

#!/bin/sh
echo `date`>>/home/vamshi/Desktop/vvvshut.txt
exit 0

the script should begin with #!/bin/sh called as "shebang" line

file is saved as
  • file.sh in /etc/init.d/
  • K10file.sh in /etc/rc0.d/
  • K10file.sh in /etc/rc6.d/
then executable permissions are given in all directories with command

chmod +x file.sh

that's it, Script executes on shutdown.


Running Script at time of startup
=======================

Script is same, but now i need to execute at startup and may be write to a different file about the startup time.

For this just put that script in /etc/rc.local file.

This file will execute at the time of startup and make that script
run for you

Mindblowing

http://blog.insicdesigns.com/2010/02/mind-blowing-javascript-experiments/

Saturday, February 6, 2010

Probe Overhead

On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0
microseconds to process. Specifically, a benchmark that hits the same
probepoint repeatedly, firing a simple handler each time, reports 1-2
million hits per second, depending on the architecture. A jprobe or
return-probe hit typically takes 50-75% longer than a kprobe hit.
When you have a return probe set on a function, adding a kprobe at
the entry to that function adds essentially no overhead.

Here are sample overhead figures (in usec) for different architectures.
k = kprobe; j = jprobe; r = return probe; kr = kprobe + return probe
on same function; jr = jprobe + return probe on same function

i386: Intel Pentium M, 1495 MHz, 2957.31 bogomips
k = 0.57 usec; j = 1.00; r = 0.92; kr = 0.99; jr = 1.40

x86_64: AMD Opteron 246, 1994 MHz, 3971.48 bogomips
k = 0.49 usec; j = 0.76; r = 0.80; kr = 0.82; jr = 1.07

ppc64: POWER5 (gr), 1656 MHz (SMT disabled, 1 virtual CPU per physical CPU)
k = 0.77 usec; j = 1.31; r = 1.26; kr = 1.45; jr = 1.9

Creating,Compiling and Loading a Kernel Module

A helloworld.c module

#include

int init_func(void)
{
printk("Hello world 1.\n");
return 0;
}
void cleanup_func(void)
{
printk("Goodbye world 1.\n");
}
module_init(init_func);
module_exit(cleanup_func);

where module_init() and module_exit are macros that label the certain functions as initializing and exiting.

Now create a Makefile

obj-m += helloworld.o

all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

if you forget to give helloworld.o and give it as helloworld.c then you will get an error saying
"target /home/vamshi/helloworld.c doesn't match the target pattern Building modules, stage 2." so be careful.

then run the command "make" for compiling the helloworld module.

Loading a Module
=============

modprobe can be used to load a module. "modprobe" is a program to add or remove modules from the kernel module.but this requires the module to be installed in /lib/modules/ DIRECTORY. Another way is to use "insmod"

insmod -- Installs a loadable kernel module in running linux kernel.

so to load the helloworld module run the command
# insmod helloworld.ko

After loading your module the function registered as the initial function will be run. Try running the dmesg command. You should see a long message ending similar to the following:

# dmesg
...
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 144k freed
eth0: link up
Hello world!

To get the list of currently loaded modules we can use the command "lsmod"

"modinfo" is a command to print out the information about one or more modules.

"rmmod" this command is to unload a module.
#rmmod helloworld.ko

Kernel Module

What is a kernel module?
A kernel module is code that can be dynamically loaded into the kernel during runtime. This makes the Linux kernel extensible so that not all functionality has to be compiled into the kernel during the initial creation. From our experience making our own system call, we know that compiling a kernel is a lengthy process and that was a SMALL kernel.
The first purpose of modules you might think of is probably device drivers that allow the kernel to interact with a new type of hardware. However, modules can add a host of other functionality like filesystems, network protocols, cryptographic libraries, etc.

dmesg

From Wikipedia, the free encyclopedia

dmesg (for "display message") is a command on some Unix-like operating systems that prints the message buffer of the kernel.

When the computer system is initially booted the kernel is loaded into memory. At this stage each device driver present in the kernel probes the system for the existence of relevant hardware. If the hardware is located, a diagnostic message is produced documenting precisely what was found. Other elements within the kernel may also produce similar output reporting both the presence of that particular module, and the values of any parameters adopted. It may be possible to specify boot parameters which control the level of detail in the messages. This process typically happens at a speed where individual messages scroll off the top of the screen before they can be read. Some keyboard keys may pause the screen output. The dmesg command allows these messages to be reviewed in a controlled manner after the system has started.

Even after the system has fully booted, the kernel may occasionally produce further diagnostic messages. Common examples of when this might happen are when I/O devices encounter errors, or USB devices are hot-plugged. dmesg provides a mechanism to review these messages at a later time. When first produced they will be directed to the system console: if the console is in use then these messages may be confused with or quickly overwritten by the output of user programs.

The output of dmesg can amount to several complete screens. For this reason, this output is normally reviewed using standard text-manipulation tools such as more, tail, or grep. The output is often captured in a permanent system logfile via a logging daemon, such as syslog. On Linux systems, similar information is sometimes found in log files in /var/log.

Many commercial operating systems display an animated splash screen during this stage of the boot process, so the user does not see these messages. However, there is frequently a mechanism, such as the Esc key, to disable the splash screen and view the messages. This is an important diagnostic capability if the system fails to boot. There is also usually a method of reviewing these messages subsequent to start up in a manner equivalent to dmesg.


Syslog

From Wikipedia, the free encyclopedia

Jump to: navigation, search

Syslog is a standard for forwarding log messages in an Internet Protocol (IP) computer network.

Syslog is a client/server protocol:[1] a logging application transmit a small (less than 1KB) text message to the syslog receiver. The receiver is commonly called syslogd, syslog daemon or syslog server. Syslog messages may be sent via the User Datagram Protocol (UDP) or the Transmission Control Protocol (TCP).[2] The data is sent in cleartext; although not part of the syslog protocol itself, an SSL wrapper may be used to provide for a layer of encryption through SSL/TLS. Syslog uses the port number 514.

Syslog is typically used for computer system management and security auditing. While it has a number of shortcomings, syslog is supported by a wide variety of devices and receivers across multiple platforms. Because of this, syslog can be used to integrate log data from many different types of systems into a central repository.

Syslog is now standardized within the Syslog working group of the IETF.

Tuesday, January 26, 2010

Procfs-Process Filesystem

The Linux implementation of /proc also clones that of Plan 9. Under Linux, /proc includes a directory for each running process (including kernel processes) at /proc/PID, containing information about that process, notably including:

  • /proc/PID/cmdline, which contains the command which originally started the process.
  • /proc/PID/cwd, a symlink to the current working directory of the process.
  • /proc/PID/environ, a file containing the names and contents of the environment variables that affect the process.
  • /proc/PID/exe, a symlink to the original executable file, if it still exists (a process may continue running after its original executable has been deleted or replaced).
  • /proc/PID/fd, a directory containing a symbolic link for each open file descriptor.
  • /proc/PID/root, a symlink to the root path as seen by the process. For most processes this will be a link to / unless the process is running in a chroot jail.
  • /proc/PID/status, a file containing basic information about a process including its run state and memory usage.
  • /proc/PID/task, a directory containing hard links to any tasks that have been started by this (i.e.: the parent) process.
  • /proc/PID/maps, the memory map showing which addresses currently visible to that process are mapped to which regions in RAM or to files.

It also includes non-process-related system information, although in the 2.6 kernel much of that information moved to a separate pseudo-file system, sysfs, mounted under /sys:

  • depending on the mode of power management (if at all), either directory, /proc/acpi or /proc/apm, which predate sysfs and contain various bits of information about the state of power management.
  • /proc/bus, containing directories representing various buses on the computer, such as input/PCI/USB. This has been largely superseded by sysfs under /sys/bus which is far more informative.
  • /proc/fb, a list of the available framebuffers
  • /proc/cmdline, giving the boot options passed to the kernel
  • /proc/cpuinfo, containing information about the CPU, such as its vendor (and CPU family, model and model names which should allow users to identify the CPU) and its speed (CPU clockspeed), cache size, number of siblings, cores, and CPU flags. It contains a value called "bogomips", frequently misunderstood as measure CPU-speed like a benchmark, while it doesn't actually measure any sensible (for end-users) value at all. It occurs as a side-effect of kernel timer calibration and yields highly varying values depending on CPU type, even at equal clock speeds.
  • /proc/crypto, a list of available cryptographic modules
  • /proc/devices, a list of character and block devices sorted by device ID but giving the major part of the /dev name too
  • /proc/diskstats, giving some information (including device numbers) for each of the logical disk devices
  • /proc/filesystems, a list of the file systems supported by the kernel at the time of listing
  • /proc/interrupts, /proc/iomem, /proc/ioports and the directory /proc/irq, giving some self-explanatory details about the devices (physical or logical) using the various system resources
  • /proc/meminfo, containing a summary of how the kernel is managing its memory.
  • /proc/modules, one of the most important files in /proc, containing a list of the kernel modules currently loaded . It gives some indication ( not always entirely correct) of dependencies.
  • /proc/mounts, a symlink to self/mounts which contains a list of the currently mounted devices and their mount points (and which file system is in use and what mount options are in use).
  • /proc/net, a directory containing a lot of really useful information about the network stack, in particular nf_conntrack which lists existing network connections (particularly useful for tracking routing when iptables FORWARD is used to redirect network connections).
  • /proc/partitions, a list of the device-numbers, their size and /dev names which the kernel has identified as existing partitions (for example if /dev/sda contains a partition table, then /dev/sda1 and others will appear as available partitions). Note that if a partition isn't listed in this file, then a patched version of losetup is around which can essentially mount the partition and connect /dev/loop[n] devices to the various partitions (though it is not certain if these will then appear in /proc/partitions).
  • /proc/scsi, giving information about any devices connected via a SCSI or RAID controller
  • a symbolic link to the current (traversing) process at /proc/self (i.e. /proc/PID/ where PID is that of the current process).
  • /proc/slabinfo, listing statistics on the caches for frequently-used objects in the Linux kernel
  • /proc/swaps, a list of the active swap partitions, their various sizes and priorities
  • Access to dynamically-configurable kernel options under /proc/sys. Under /proc/sys appear directories representing the areas of kernel, containing readable and writable virtual files.
    For example, a commonly referenced virtual file is /proc/sys/net/ipv4/ip_forward, because it is necessary for routing firewalls or tunnels. The file contains either a '1' or a '0': if it is 1 then the IPv4 stack will forward packets not meant for the local host, if it is 0 then it will not.
  • /proc/sysvipc, containing memory sharing and IPC information.
  • /proc/tty, containing information about the current terminals; /proc/tty/driver looks to be a list of the different types of tty available each of which is a list of those of each type
  • /proc/uptime, the length of time the kernel has been running since boot and spent in idle mode (both in seconds)
  • /proc/version, containing the Linux kernel version, distribution number, gcc version number (used to build the kernel) and any other pertinent information relating to the version of the kernel currently running
  • other files depending on various hardware, module configurations, and changes to the kernel.

The basic utilities that use /proc under Linux come in the procps (/proc processes) package, and only function in conjunction with a mounted /proc.

The procfs plays an important role in moving functionality from kernel space to user space. For example the GNU version of ps operates entirely in user mode, using the procfs to obtain its data.

Sunday, January 24, 2010

File System Hierarchy (FHS)

Directory Description
/ Primary hierarchy root and root directory of the entire file system hierarchy.
/bin/ Essential command binaries that need to be available in single user mode; for all users, e.g., cat, ls, cp.
/boot/ Boot loader files, e.g., kernels, initrd; often a separate partition[8]
/dev/ Essential devices, e.g., /dev/null.
/etc/ Host-specific system-wide configuration files (the name comes from et cetera[9]).
/etc/opt/
Configuration files for /opt/.
/etc/X11/
Configuration files for the X Window System, version 11.
/etc/sgml/
Configuration files for SGML.
/etc/xml/
Configuration files for XML.
/home/ Users' home directories, containing saved files, personal settings, etc.; often a separate partition.
/lib/ Libraries essential for the binaries in /bin/ and /sbin/.
/media/ Mount points for removable media such as CD-ROMs (appeared in FHS-2.3).
/mnt/ Temporarily mounted filesystems.
/opt/ Optional application software packages[10].
/proc/ Virtual filesystem documenting kernel and process status as text files, e.g., uptime, network. In Linux, corresponds to a Procfs mount.
/root/ Home directory for the root user.
/sbin/ Essential system binaries, e.g., init, route, mount.
/srv/ Site-specific data which is served by the system.
/tmp/ Temporary files (see also /var/tmp). Often not preserved between system reboots.
/usr/ Secondary hierarchy for read-only user data; contains the majority of (multi-)user utilities and applications.[11]
/usr/bin/
Non-essential command binaries (not needed in single user mode); for all users.
/usr/include/
Standard include files.
/usr/lib/
Libraries for the binaries in /usr/bin/ and /usr/sbin/.
/usr/sbin/
Non-essential system binaries, e.g., daemons for various network-services.
/usr/share/
Architecture-independent (shared) data.
/usr/src/
Source code, e.g., the kernel source code with its header files.
/usr/X11R6/
X Window System, Version 11, Release 6.
/usr/local/
Tertiary hierarchy for local data, specific to this host. Typically has further subdirectories, e.g., bin/, lib/, share/.[12]
/var/ Variable files—files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files. Sometimes a separate partition.
/var/lib/
State information. Persistent data modified by programs as they run, e.g., databases, packaging system metadata, etc.
/var/lock/
Lock files. Files keeping track of resources currently in use.
/var/log/
Log files. Various logs.
/var/mail/
Users' mailboxes.
/var/run/
Information about the running system since last boot, e.g., currently logged-in users and running daemons.
/var/spool/
Spool for tasks waiting to be processed, e.g., print queues and unread mail.
/var/spool/mail/
Deprecated location for users' mailboxes.
/var/tmp/
Temporary files to be preserved between reboots.

Wednesday, January 20, 2010

Some Interesting links i found

Source of Name Generators:

http://www.ruf.rice.edu/~pound/

LinkSwarm Intelligence in bird flocks

http://webjazz.blogspot.com/search/label/ruby
Link
Source Code:http://github.com/iamwilhelm/frock

Python Script for webparser
http://www.daniweb.com/forums/thread161803.html#

http://www.cyberciti.biz/tips/howto-shutdown-linux-box-automatically.html
Link

http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html

http://www.cyberciti.biz/tips/howto-write-system-utilization-data-to-file.html

http://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html
Link

http://www.cyberciti.biz/tips/linux-strace-command-examples.html
Link

http://www.cyberciti.biz/files/linux-kernel/Documentation/filesystems/proc.txt

How To Write A Name Generator (In Ruby)

Refer: http://www.skorks.com/2009/07/how-to-write-a-name-generator-in-ruby/

Link

Installing And Using SQLite With Ruby On Windows

SQLite is a great little embeddable database engine that’s meant to be self-contained, easy to use and not require configuration. However when I tried to use it in my Ruby code I found that it wasn’t that straight forward getting everything to work, especially if you’re using Ruby on windows. The information on the old WWW was rather sparse. I worked it out eventually, so in the interests of being a helpful web denizen I thought I’d share.

What To Download And Where To Put It

First thing first, before we involve Ruby directly lets download set up everything we need. You need to download two things from the SQLite website (well strictly speaking you only actually NEED one, but we’ll download both since the other one is useful):

  • sqlitedll-3_6_16.zip – you can think of this one as actually being the SQLite database that you will install on your machine
  • sqlite-3_6_16.zip – this one is a command line utility that can be used to administer a SQLite database

Both of those files are located on the downloads page – http://www.sqlite.org/download.html, under the ‘Precompiled Binaries For Windows’ section.

Once you unzip both of those you will have several files, the important ones are:

  • sqlite3.dll – this is the actual database
  • sqlite3.exe – this is the command line utility

You need to take the DLL and put it somewhere, where Ruby would look for libraries when you execute a script, some possible locations would be, your windows system folder or the bin folder of your Ruby installation (Note: as reader Luis Lavena noted in the comments below, the ruby installation bin folder is by far the best place to put the DLL). On my machine it was:

C:\ruby1.8\bin
C:\WINDOWS\system
C:\WINDOWS\system32

You can also put the command line utility in any of those places as well, but you don’t have to as you can use that one from anywhere (if you do put it in another location make sure that location is in the PATH environment variable otherwise you will have to specify the full path to it every time you want to use it).

You’re now ready to install the SQLite Ruby interface gem. You will need to do the following:

gem install sqlite3-ruby

After it completes you will have a gem similar to the following in your gem repository, in my case:

C:\ruby1.8\lib\ruby\gems\1.8\gems\sqlite3-ruby-1.2.5-x86-mswin32

You’re now ready to work with SQLite in your Ruby application. Note that you need to install the ‘sqlite3-ruby’ gem and not the ‘sqlite-ruby’ one, as that is the old one.

If you did not install the DLL correctly you might get an error similar to the following when you try to run your ruby script:

C:/ruby1.8/lib/ruby/1.8/dl/import.rb:29:in `initialize': unknown error (RuntimeError)
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:29:in `dlopen'
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:29:in `dlload'
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:27:in `each'
from C:/ruby1.8/lib/ruby/1.8/dl/import.rb:27:in `dlload'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/driver/dl/api.rb:31
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/driver/dl/driver.rb:1
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from C:/ruby1.8/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:619:in `load_driver'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:617:in `each'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:617:in `load_driver'
from C:/ruby1.8/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.2.5-x86-mswin32/lib/sqlite3/database.rb:72:in `initialize'
from D:/ruby-projects/search-engine/search-engine/lib/search-engine-main.rb:21:in `new'
from D:/ruby-projects/search-engine/search-engine/lib/search-engine-main.rb:21

This is because the SQLite ruby interface is trying to use a deprecated driver (it’s called DL and it is buggy and doesn’t work) because it can’t find the native one, you need to make sure you put the DLL in the appropriate spot mentioned above. Also note that putting the DLL in the same directory as the script you’re executing does not work and produces a different error possibly because it has a dependency on MSVCRT.DLL.

How To Use SQLite With Ruby

We can now write some ruby code to test that everything is working, as well as learn to use SQLite a little bit.

require 'sqlite3'
database = SQLite3::Database.new( "new.database" )

database.execute( "create table sample_table (id INTEGER PRIMARY KEY, sample_text TEXT, sample_number NUMERIC);" )

database.execute( "insert into sample_table (sample_text,sample_number) values ('Sample Text1', 123)")
database.execute( "insert into sample_table (sample_text,sample_number) values ('Sample Text2', 456)")

rows = database.execute( "select * from sample_table" )

p rows

Running this code will create a new file called ‘new.database’ in the same folder where you ran the script. This is the sqlite database you’ve just created (sqlite stores data as files). To execute any SQL code using sqlite, you simply call the ‘execute’ method on your database instance and provide it some SQL code as a string. This is what we did to create a table and to insert two rows into it. Our little script produces the following output:

[["1", "Sample Text1", "123.0"], ["2", "Sample Text2", "456.0"]]

Which shows that we can get the two rows that we inserted back from the database.

If you attempt to re-run this script, it will simply pick up the database file since the file persists past the execution of the script. Incidentally the script would fail a second time since the table ‘sample_table’ already exists in the database.

You now have the basics down, from here the going is a little easier. You can look at the following resources:

SQLite/Ruby FAQ – all the different ways to use sqlite from a Ruby script

SQLite Documentation – self explanatory

Supported SQL Syntax – self explanatory

SQLite Datatypes – how datatypes work in SQLite 3

Enjoy!