Sysadmin Geek

Upload Files to an FTP Site via a Batch Script

Jason Faulkner — Wed, 08 Sep 2010 10:00:42 +0000

Outside of email, probably the most common way to send files to a remote party is via FTP. While there are a plethora of FTP clients you can choose from, Windows has an little known and under utilized command line FTP utility built in. The beauty of this tool lies in it’s ability to be scripted which we have harnessed in the batch script below.

This script can be used from the command line as a ‘no questions asked’ method of uploading one or many files with a single command. Additionally, you can call this script from batch files to perform automated file uploads. A few uses for this include (but, of course, not limited to):

Include in backup scripts to send data offsite.
Upload html/php/etc. files to a web server with a single command.
Create shortcuts to send a common group of files (such as a web site’s source pages).

Configuration

The only configuration required is to set the FTP server connection information. Under the “Connection information” line, set the following:

Server – The FTP Server you are uploading to. You can either enter the DNS name (ftp.myserver.com) or IP address (1.2.3.4).
UserName – Your user name for connecting to FTP server.
Password – Your password for connecting to the FTP server.

Depending on your firewall settings, the first time you run this script you may be prompted to allow FTP to connect to the Internet. Setting this to never prompt you again should remove future warnings.

The Script

@ECHO OFF
ECHO Upload to FTP
ECHO Written by: Jason Faulkner
ECHO SysadminGeek.com
ECHO.
ECHO.

REM Usage:
REM UploadToFTP [/L] FileToUpload
REM
REM Required Parameters:
REM  FileToUpload
REM      The file or file containing the list of files to be uploaded.
REM
REM Optional Parameters:
REM  /L  When supplied, the FileToUpload is read as a list of files to be uploaded.
REM      A list of files should be a plain text file which has a single file on each line.
REM      Files listed in this file must specify the full path and be quoted where appropriate.

SETLOCAL EnableExtensions

REM Connection information:
SET Server=
SET UserName=
SET Password=

REM ---- Do not modify anything below this line ----

SET Commands="%TEMP%\SendToFTP_commands.txt"

REM FTP user name and password. No spaces after either.
ECHO %UserName%> %Commands%
ECHO %Password%>> %Commands%

REM FTP transfer settings.
ECHO binary >> %Commands%

IF /I {%1}=={/L} (
   REM Add file(s) to the list to be FTP'ed.
   FOR /F "usebackq tokens=*" %%I IN ("%~dpnx2") DO ECHO put %%I >> %Commands%
) ELSE (
   ECHO put "%~dpnx1" >> %Commands%
)

REM Close the FTP connection.
ECHO close  >> %Commands%
ECHO bye    >> %Commands%

REM Perform the FTP.
FTP -d -i -s:%Commands% %Server%

ECHO.
ECHO.

REM Clean up.
IF EXIST %Commands% DEL %Commands%

ENDLOCAL

Links

Download Upload to FTP Script from Sysadmin Geek

Automating the Process of Deleting Old Log Files

Jason Faulkner — Tue, 07 Sep 2010 10:00:36 +0000

Many services and programs out there produce log files as an audit trail for everything they are doing, however few have a function which removes these files as they outlive their usefulness. As a result, these log files sit on your system eating up space (sometimes more than you know) and cluttering directories for those times you need to access them.

So if you don’t need these files, why keep them? We are going to show you how to easily remove these old log files to keep you system nice and tidy.

Of course, while the we are covering below are immediately useful for managing log files, you can also apply the same techniques to any other type of “expiring” file (such as backups).

Remove Files Based on Last Modified Date

If you want to clear your existing log files based solely on the last modified date of the file, all you have to do is use the FORFILES command. For example:

FORFILES /P “C:\LogFiles” /S /D -7 /C “CMD /C DEL /F /Q @PATH”

The above command would delete all files from the “C:\LogFiles” folder, and all sub-folders which have not been modified in the last week.

The FORFILES command is pretty flexible with the search pattern and date functions. For example, in place of a number, you can enter a date such as ‘-1/13/2010’ to delete files last modified prior to the specified date.

To get all the details on what FORFILES can do, view the online help using the following command from the command prompt:

FORFILES /?

Remove Files Based on a Date Pattern in the File Name

Many applications and services produce log files based on a date pattern as to have one log file per day (i.e. Log100113.txt, Backup-2010-01-13.zip, etc.). For these types of files, it is preferable to delete based on the date of the file incorporated into the file name rather than the last modified date. This is useful for scenarios such as keeping all log files for the past 3 months. Unfortunately, Windows does not have a native command with this type of logic but with a batch script we can easily handle this task.

There are examples included in the usage comments on the script, so it should be pretty easy to figure out.

The Script

@ECHO OFF
ECHO Delete By Date Pattern
ECHO Written by: Jason Faulkner
ECHO SysadminGeek.com
ECHO.
ECHO.

REM Delete/Select files based on a date which utilizes MM and/or DD for file naming patterns.
REM
REM Usage:
REM DeleteByDatePattern {/M | /D} NumberToKeep Path PatternPrefix PatternPostfix [/L | /DEL]
REM    /M     Specifies the pattern being used is based on months.
REM    /D     Specifies the pattern being used is based on days.
REM    NumberToKeep
REM           The number of months (/M) or days (/D) to keep, including the current.
REM           For example, entering 1 keeps only the current month/day and 6 would keep the current minus 5.
REM    Path   The root location to search. Subdirectories will be searched.
REM    PatternPrefix
REM           The file search pattern placed before of the month/day when building the search string.
REM    PatternPostfix
REM           The file search pattern placed after of the month/day when building the search string.
REM    /L     (optional) Lists all files matching the pattern, but does not delete them.
REM    /DEL   (optional) Deletes all files matching the pattern.
REM
REM Examples:
REM    DeleteByDatePattern /M 3 "%WinDir%\system32\LogFiles" ex?? ??.log /DEL
REM       Deletes all IIS log files (Windows Server 2003) except for the current and previous two months.
REM    DeleteByDatePattern /D 7 "D:\Backup" *-????-??- .zip /DEL
REM       Deletes all zip files from the D:\Backup folder except for the current week.
REM       The file name pattern assumed above is "*-YYYY-MM-DD.zip"
REM    DeleteByDatePattern /M 0 "C:\" *( )* /L
REM       Prints a list of all files on the C drive matching the pattern: "*-MM-*" (where MM is replaced with 01-12)
REM    DeleteByDatePattern /D 14 "C:\Logs" Log-???? .txt
REM       Prints a list of all patterns which would be processed by the script.

SETLOCAL EnableExtensions EnableDelayedExpansion

REM Assumes your Windows Date/Time settings are set to 'DayOfWeek M/D/YYYY' format.
REM If your format is different, you will need to alter the variables below so they align.
FOR /F "tokens=1,2,3,4 delims=/ " %%A IN ('DATE /T') DO (
   SET Month=%%B
   SET Day=%%C
   SET Year=%%D
)

IF /I {%1}=={/M} (
   SET Keep=%Month%
   SET Max=12
)
IF /I {%1}=={/D} (
   SET Keep=%Day%
   SET Max=31
   REM Working off of the previous month's max days.
   SET /A PrevMonth=%Month%-1
   IF !PrevMonth! EQU 2 (
      SET Max=28
      REM Leap years... add more as needed.
      IF /I %Year% EQU 2012 SET Max=29
      IF /I %Year% EQU 2016 SET Max=29
   )
   IF /I !PrevMonth! EQU 4 SET Max=30
   IF /I !PrevMonth! EQU 6 SET Max=30
   IF /I !PrevMonth! EQU 9 SET Max=30
   IF /I !PrevMonth! EQU 11 SET Max=30
)
SET Current=%Keep%
SET /A Keep=%Keep%-%2+1

REM Determine the range to be removed.
SET /A RemoveHighStart=%Current%+1
IF /I %Keep% LSS 1 (
   SET RemoveLow=0
   SET /A RemoveHighEnd=%Keep%+%Max%-1
) ELSE (
   SET /A RemoveLow=%Keep%-1
   SET RemoveHighEnd=%Max%
)

REM Process all less than the low range.
FOR /L %%Z IN (1,1,%RemoveLow%) DO CALL  rocess %%Z %3 %4 %5 %6
REM Process all greater than the high range.
FOR /L %%Z IN (%RemoveHighStart%,1,%RemoveHighEnd%) DO CALL  rocess %%Z %3 %4 %5 %6

ENDLOCAL
GOTO End

  rocess
SET Key=0%1
SET Key=%Key:~-2%
SET Target="%~2\%~3%Key%%~4"
ECHO Target Pattern: %Target%

IF /I {%5}=={/L} DIR %Target% /B /S
IF /I {%5}=={/DEL} DEL /F /S /Q %Target%
GOTO End

:End

Automating the Process

The FORFILES command is native to Windows, however the DeleteByDatePattern script should be placed in a folder defined in your Path variable (such as your Windows folder) so it can be called as though it were a native command. Once this is done, you can create a scheduled task which is either a single command (if you only need to delete from a single location) or a batch file (if you need to delete from multiple locations) which runs daily, weekly, monthly or whenever.

One more thing you can set and forget.

Links

Download Delete By Date Pattern script from Sysadmin Geek

Automated Strategies for Managing Outdated Backup Files

Jason Faulkner — Mon, 06 Sep 2010 10:00:51 +0000

Backups are something which, usually, run on a very frequent schedule. If left unmanaged, a direct result of this is a large number files eating up a potentially large amount of hard drive space. Remembering to manually go in and remove backup files certainly is one method of management but shouldn’t be a long term strategy, especially when there are easy to implement automated procedures available.

There is no ‘one size fits all’ solution for automating the process of deleting expired backup files. Depending on the backup procedure (do you do weekly full and daily incrementals or daily full?) and file naming convention (does your backup file have the date as part of the file name or use the same file name every time?) of your resulting backup files, the applicable method differs.

We are going to point out several simple solutions which fit the most common backup scenarios, so odds are one will be right for you.

Deleting by File Age or Date

When to use: Daily full backups.

Perhaps the simplest and most logical way to purge expired backups is to base the deletion process on the date of the backup file. We have previously covered this topic using log files as our target files, however it works just as well with backup files.

For example, to delete any files in the specified folder not modified within the past week, run this command:

FORFILES /P “C:\Backups” /S /D -7 /C “CMD /C DEL /F /Q @PATH”

Note the keyword above: modified. The ForFiles command is only capable of evaluating the modified file date as opposed to the created date which would be more applicable. Typically, however, you are probably not modifying a backup file after it is created so this will most likely not be an issue.

Alternately, if your backup file has some sort of numeric date pattern specified in the file name (i.e. Backup_2010-01-13.zip, BackupSet_100113_Full.zip, etc.), you can use the DeleteByDatePattern script we provided in the linked article to remove expired backups.

For example, to delete files older than 2 weeks matching a file name pattern like the following: “Backup_YYYY-MM-DD_(Full | Incremental).zip”, you would use the command:

DeleteByDatePattern /D 15 “C:\Backups” *-????-??- _*.zip /DEL

Or if your file naming pattern is: “BackupSet_YYMMDD.zip”, you would use:

DeleteByDatePattern /D 15 “C:\Backups” *-???? .zip /DEL

Of course, adjust as needed but either of the methods above could easily be added to the start or end of your backup process to keep the number of backups stored manageable.

Folder Rolling

When to use: Periodic full backups (weekly, bi-weekly, etc.) with daily incremental backups in between.

The idea behind “folder rolling” is that you store all of your current backup set (full backup + respective incrementals) in a single folder and then have several archive folders where your old backup sets are kept. Prior to a new backup set being created, you delete the folder contents containing oldest backup set and “roll” the contents of each folder down one.

For example, suppose we have a current backup folder with two archive folders. The batch script commands to perform the folder roll for this would be:

DEL /F /Q “C:\Backups\2archive”
MOVE /Y “C:\Backups\1archive\*” “C:\Backups\2archive”
MOVE /Y “C:\Backupscurrent\*” “C:\Backups\1archive”

You can add as many archive folders as needed. Just delete the contents of the lowest archive folder add a move command for each of the other archive folders.

Again, this works best for situations where you create a periodic full backup and the a number of incremental backups up until your next full backup. Simply drop all your related backup files into a single folder and the run the folder roll script right before you create a new backup set.

Backup9

When to use: Daily full backups or individual file backups.

Backup9 is a free command line utility developed by Gammadyne. Similar to the folder rolling process above, the idea behind this utility is simple in that when it is run, a copy of the target file is created with a number appended to the end. Additionally, you specify a cut-off of the number of copies to keep with the default being 9 (hence the name).

An example will best explain this process. Using the following command would produce the output below:

BACKUP9 /A /L7 “C:\Backups\BackupFile.zip”

If this command were run again, the following would happen:

The number of files to keep (7 in our example) is evaluated and if there are currently that many copies, the last one is dropped.
BackupFile.zip.bk7 is deleted.
BackupFile.zip.bk6 is renamed to BackupFile.zip.bk7
BackupFile.zip.bk[#] is renamed to BackupFile.zip.bk[#+1]
BackupFile.zip.bk1 is renamed to BackupFile.zip.bk2
BackupFile.zip is copied and named BackupFile.zip.bk1

With the ability to keep up to 999 copies, this utility works very well if you have a file with a static name. You simply add the Backup9 command to the beginning or end of your backup process it takes care of keeping up with the appropriate number of archive copies.

Belvedere Automated File Manager

When to use: Daily full backups.

Belvedere automated file manager is a utility which runs in the background monitoring file system active and performs configured actions when specified conditions are met. Among its many uses are cleaning up expired backup files.

The configuration of the rules are pretty straightforward. For example, to create a rule to delete backup files using a file name pattern such as “BackupSet_Jan13.zip” which are older than 2 weeks, you could use the following:

While the basic function we are performing can easily be done with command line tools described above, the obvious difference is that Belvedere provides an easy to use graphical interface for those more comfortable with pointing and clicking.

Belvedere is designed as a desktop user application which runs from the system tray, however you can run Belvedere as a service and use it on servers to perform this and other file monitoring operations.

Conclusion

While there are a myriad number of ways you can manage your backup expiration process, the methods we have described above are both flexible and easy to implement. With a bit of experimentation, find what works for you and go with it so you can just set it and forget it.

Links

Download Backup9 from Gammadyne.com

Download Belvedere from Lifehacker.com

How to Setup milter-greylist Spam Blocking in Sendmail

Thom Jones — Fri, 03 Sep 2010 10:00:05 +0000

Virtually all linux distributions include sendmail as the default MTA. Which is okay – it has been around for a long time, is stable and it works great (although the postfix afficionados might disagree!). But it has nothing built in for spam control which is good; it was not designed for that. So you’ve installed spamassassin and it works good but you still are getting unflagged spam emails through. Perhaps you need to try greylisting.

Greylisting is the process by which all email (unless specifically whitelisted) gets initially rejected yet works within the parameters of the various RFCs to ensure eventual receipt of email. The idea is that spammers will not attempt to reconnect to an email server that has rejected their offerings yet legitimate mail servers will. It isn’t foolproof – spammers are quick to adjust and greylisting has been around a long time. But it does help.

This article is on how to install milter-greylist which was originally written by Emmanuel Dreyfus. I will be concentrating with sendmail here but milter-greylist is also supported with postfix.

First, check your dependencies. From the README:

Build dependencies:
- flex (AT&T lex cannot build milter-greylist sources)
- yacc or bison (some older yacc will fail, use bison instead)
- libmilter (comes with Sendmail, or with the sendmail-devel
package on RedHat, Fedora and SuSE. Debian and Ubuntu have it
in libmilter-dev)
- Any POSIX threads library (Provided by libc on some systems)

Optional dependencies:
- libspf2, libspf_alt or libspf, for SPF support
- libcurl, for URL checks support
- libGeoIP, for GeoIP support
- libbind from BIND 9, for DNSRBL support, except if your system has a thread-safe DNS resolver built-in.

But the configuration process will find anything that you don’t have installed and complain until the dependency is resolved.

Next, download the greylist-milter from http://hcpnet.free.fr/milter-greylist and unpack the tarball. Then read the README file! It includes a wealth of information that isn’t covered in this article especially for installs that want/need to include special features such as SPF support.

And do the usual

./configure
./make
./make install

The standard install will put the binaries in /usr/local/bin, the database and pid file in /var/milter-greylist and the configuration file will be /etc/mail/greylist.conf. Some startup scripts are included in the tarball but they are not installed automatically. You will have to set it up in your /etc/init.d yourself if you want to use one.

You will then need to configure sendmail to actually use the milter. In your sendmail.mc file, add the following (but pay close attention to the warnings in the README file if you are already using other milters in your installation!):

INPUT_MAIL_FILTER(`greylist',`S=local:/var/milter-greylist/milter-greylist.sock')dnl define(`confMILTER_MACROS_CONNECT', `j, {if_addr}')dnl define(`confMILTER_MACROS_HELO', `{verify}, {cert_subject}')dnl define(`confMILTER_MACROS_ENVFROM', `i, {auth_authen}')dnl define(`confMILTER_MACROS_ENVRCPT', `{greylist}')dnl

and reconfigure your sendmail.cf file:

#m4 sendmail.mc > sendmail.cf

Don’t restart your sendmail daemon just yet, however – we still have to modify the configuration for this to work properly.

Open /etc/mail/greylist.conf in your favorite editor (which, of course, is vi, right?).
Uncomment or add the following:

quiet greylist 7m dumpfreq 1d autowhite 10d

In the above configuration, ‘quiet’ will not include a time frame to retry submission. This is good so that there is no way for spammers to know how long they will be blocked. Greylisting will be for 7 minutes after which email from the source will be accepted, database contents will be dumped to the /var/milter-greylist/greylist.db once per day, and, once an email is accepted from a source, that source will be whitelisted for 10 days before being greylisted again.

Also create lists to whitelist your own networks to the configuration file:

list "my network" addr { 127.0.0.1/8 10.230.1.0/24 192.168.1.0/24 }

which will whitelist local, DMZ and internal networks (as an example – yours are probably different). Notice the space between network addresses, not commas.
Along with other external networks that are always trusted:

# Trusted networks to not greylist: list "trusted" addr { \ 207.46.0.0/16 \ # Microsoft 72.33.0.0/16 \ # UW Madison }

There is a fairly comprehensive list of ‘broken’ mailer servers in the configuration file that are also always to be whitelisted since greylisting them would most likely result in never getting email from them. You can add to that list as needed as well if you need to.

You will most likely be setting up greylisting as the default, so you may also want to whitelist certain users who never want email to be delayed (various pompous vice presidents, system alert addresses and the like):

# List of users that want whitelisting (if greylisting is the default): list "white users" rcpt { \ vp@domain.com \ sysadmin@domain.com \ postmaster@domain.com \ }

Notice the list names of “my network”, “trusted” and “white users” – you need to add these to the actual whitelisting config line:

# And here is the access list racl whitelist list "my network" racl whitelist list "broken mta" racl whitelist list "trusted" racl whitelist list "white users"

Note: You can also set this up to whitelist as the default in which case you would also create a “grey users” list of those folk you want to always be subject to greylisting. Those would include errant users who post their work email address all over social network sites, sales web sites and newsletter subscriptions, of course.

And then configure the default operation of milter-greylist:

racl greylist default

(use racl whitelist default if you want whitelisting to be the default operation).

And then fire up your milter-greylist binary either using the /etc/init.d/milter-greylist startup script or by

#milter-greylist -f /etc/mail/greylist.config

at the command line. There are a slew of other command line options (many of which duplicate parameters set in the conf file). See

man milter-greylist

for further details.

And then restart your sendmail daemon and enjoy less spam coming into your mail server.

How to Prevent Yum from Updating the Kernel

Thom Jones — Thu, 02 Sep 2010 10:00:13 +0000

When you’re running production servers, the one thing you don’t want to do is upgrade the kernel every time a new update comes out. Why? Because that’s the only Linux update operation that requires a reboot once it’s done—and in a production environment you often can’t have downtime.

The Scenario

So you finally have your rpm based server set, stable and secure. Life is good and you don’t figure you have to do anything else for some time.

Then, for some reason, you run at the command line as root:

#yum -y -d0 update

Which just happens to supply an affirmative answer to all prompts for the yum command. Which also happens to run as quietly as possible. And which happens this time to include an update to the kernel packages.

And now things are not working correctly. How could you have prevented this?

Although you obviously and desperately need a vacation now, you should revisit your system update schedule, your documentation (which, of course, is meticulous, current and readily available) and especially the configuration files that control yum.

But first, why wouldn’t you want to update the kernel?

Things sometimes break. An update might make modules or drivers incompatible so devices such as a wireless card are no longer functional.
Maintain versions across server populations. This would certainly apply if you have a mix of different server distributions along with some home built boxes.
Compiled the kernel manually, thus bypassing the automatic configuration supplied by yum
You are horribly proud of your uptime so performing a reboot to activate the new kernel would not be in your best interest. So if you never reboot, there is never a need to update it.

Why would you want to update the kernel?

Primarily – security. Kernels have holes just like applications and, if not patched, can provide opportunities for the compromising of the system by the bad guys.
Especially if you are not using a 3rd party repository, to keep your version as up to date as possible. Major distros release patches to all of their included code including the kernel and strive to test it before release to ensure stability. To not update is to lag behind which might make things more difficult when performing an upgrade to a major version release.
Taking advantage of a new feature (or to fix a previously broken one) would make you possibly more amenable to a kernel update.

To update the kernel just do what you did above. Or, better, eliminate the command line switches so you have some control over the update process and can see what is happening. Run:

#yum update

and follow the prompts.

Preventing Yum from Updating the Kernel

However, if you don’t ever want to just blindly have the kernel updated, you can add the following to your /etc/yum.conf file:

exclude=kernel*

Or, if you insist on using a vanilla configuration file and control everything via the CLI, use

#yum –exclude=kernel* update

Both of these methods will eliminate the kernel from being updated or even included in the potential update listing.

How to Change Your Linux Hostname Without Rebooting

Thom Jones — Wed, 01 Sep 2010 10:00:44 +0000

If you’ve worked in the admin world for any length of time, you’ve probably run into an instance where you needed to change the hostnames on your server to match some corporate naming standard, but you can’t have downtime either. So how do you change the hostname without rebooting?

Changing the Hostname

First, you must change the config file that controls this. The actual file and its location will vary across distributions. In the Redhat derivatives, the file to modify is /etc/sysconfig/network so vi this file and change the line that reads HOSTNAME=

From this:

To this:

For SuSE distributions you would similarly change the /etc/HOSTNAME file while Debian admins would modify the /etc/hostname file to set the new name.

Next, the /etc/hosts file needs to be changed. This is identical across all flavors and consists of using your favorite text editor and modifying the old hostname within. Such as from this:

To this:

“But”, you mutter under your breath, “I just typed in `hostname` and got ‘Snoopy’ as a result. It doesn’t work!”.

What you need to know is that the previous steps are to permanently set the new name. You are much too quick to judge, obviously. So, continue on by entering (at the command line as root or a sudoer):

# hostname mdmvr14s9db

This will set the hostname for now but the change is not saved (that is what the above steps are for, remember?). Go ahead – test it. Type in `hostname` (without the tickmarks ) and the command should return ‘mdmvr14s9db’ (without the single quotes).

Now let’s set it in stone. If you are using chkconfig and service commands (RedHat family folk, usually), you can simply type

#service network restart

which will restart the network with the new host name. The rest of you should be able to type

#/etc/init.d/network restart

to obtain the same results.

Then test again with the hostname command – you should get the new name returned.

This has been the easy part. Hopefully, prior to starting this, you checked all application conf files for any hard coded reference to the new name and changed that as well. And you need to then change your DNS ‘A’ record to also reflect the new name. And once you have restarted named with the new name on the DNS you will have succeeded in temporarily irritating around half of those 18,000 users previously mentioned who now can’t get to the server since the name has not propagated around the network yet. But this too shall pass as long as you hide long enough.

And at least you can keep the small stuffed plush Snoopy sitting at your desk to remind you of that gentler, simpler time.

Diagnose Linux Server Load Problems with a Simple Script

Thom Jones — Tue, 31 Aug 2010 10:00:12 +0000

If you have been an admin for any length of time, you have certainly discovered situations where a server spikes in CPU use or memory utilization and/or load levels. Running `top` won’t always give you the answer, either. So how do you find those sneaky processes that are chewing up your system resources to be able to kill ‘em?

The following script might be able to help. It was written for a web server, so has some parts of it that are specifically looking for httpd processes and some parts that deal with MySQL. Depending on your server deployment, simply comment/delete those sections and add others. It should be used for a starting point.

Prerequisites for this version of the script is some freeware released under the GNU General Public License called mytop (available at http://jeremy.zawodny.com/mysql/mytop/) which is a fantastic tool for checking how MySQL is performing. It is getting old, but still works great for our purposes here.
Additionally, I use mutt as the mailer – you may want to change the script to simply use the linux built in `mail` utility. I run it via cron every hour; adjust as you see fit. Oh – and this script needs to run as root since it does read from some protected areas of the server.

So let’s get started, shall we?

First, set your script variables:

#!/bin/bash # # Script to check system load average levels to try to determine # what processes are taking it overly high... # # 07Jul2010 tjones # # set environment dt=`date +%d%b%Y-%X` # Obviously, change the following directories to where your log files actually are kept tmpfile="/tmp/checkSystemLoad.tmp" logfile="/tmp/checkSystemLoad.log" msgLog="/var/log/messages" mysqlLog="/var/log/mysqld.log" # the first mailstop is standard email for reports. Second one is for cell phone (with a pared down report) mailstop="sysadmin@mydomain.com" mailstop1="15555555555@mycellphone.com" machine=`hostname` # The following three are for mytop use - use a db user that has decent rights dbusr="username" dbpw="password" db="yourdatabasename" # The following is the load level to check on - 10 is really high, so you might want to lower it. levelToCheck=10

Next, check your load level to see if the script should continue:

# Set variables from system: loadLevel=`cat /proc/loadavg | awk '{print $1}'` loadLevel=$( printf "%0.f" $loadLevel )
# if the load level is greater than you want, start the script process. Otherwise, exit 0
if [ $loadLevel -gt $levelToCheck ]; then echo "" > $tmpfile echo "**************************************" >>$tmpfile echo "Date: $dt " >>$tmpfile echo "Check System Load & Processes " >>$tmpfile echo "**************************************" >>$tmpfile

And continue through the checks, writing the results to the temporary file. Add or delete items from here where applicable to your situation:

# Get more variables from system: httpdProcesses=`ps -def | grep httpd | grep -v grep | wc -l`
# Show current load level: echo "Load Level Is: $loadLevel" >>$tmpfile echo "*************************************************" >>$tmpfile # Show number of httpd processes now running (not including children): echo "Number of httpd processes now: $httpdProcesses" >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Show process list: echo "Processes now running:" >>$tmpfile ps f -ef >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile
# Show current MySQL info: echo "Results from mytop:" >>$tmpfile /usr/bin/mytop -u $dbusr -p $dbpw -b -d $db >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile

Notice with the top command, we are writing to two temp files. One is for the much smaller message to cell phone. If you don’t want the urgency of cell phone alerts at three in the morning, you can take this out (and take out the second mailing routine later in the script).

# Show current top: echo "top now shows:" >>$tmpfile echo "top now shows:" >>$topfile /usr/bin/top -b -n1 >>$tmpfile /usr/bin/top -b -n1 >>$topfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile

More checks:

# Show current connections: echo "netstat now shows:" >>$tmpfile /bin/netstat -p >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile
# Check disk space echo "disk space:" >>$tmpfile /bin/df -k >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile

Then write the temporary file contents to a more permanent log file and email the results to the appropriate parties. The second mailing is the pared down results consisting simply of the standard out of `top`:

# Send results to log file: /bin/cat $tmpfile >>$logfile
# And email results to sysadmin: /usr/bin/mutt -s "$machine has a high load level! - $dt" -a $mysqlLog -a $msgLog $mailstop <$tmpfile /usr/bin/mutt -s "$machine has a high load level! - $dt" $mailstop1 <$topfile
echo "**************************************" >>$logfile

And then some housekeeping and exit:

# And then remove the temp file: rm $tmpfile rm $topfile fi
# exit 0

Hopefully this helps someone out there. Fully assembled script is:

#!/bin/bash # # Script to check system load average levels to try to determine what processes are # taking it overly high... # # set environment dt=`date +%d%b%Y-%X` # Obviously, change the following directories to where your log files actually are kept tmpfile="/tmp/checkSystemLoad.tmp" logfile="/tmp/checkSystemLoad.log" msgLog="/var/log/messages" mysqlLog="/var/log/mysqld.log" # the first mailstop is standard email for reports. Second one is for cell phone (with a pared down report) mailstop="sysadmin@mydomain.com" mailstop1="15555555555@mycellphone.com" machine=`hostname` # The following three are for mytop use - use a db user that has decent rights dbusr="username" dbpw="password" db="yourdatabasename" # The following is the load level to check on - 10 is really high, so you might want to lower it. levelToCheck=10 # Set variables from system: loadLevel=`cat /proc/loadavg | awk '{print $1}'` loadLevel=$( printf "%0.f" $loadLevel )
# if the load level is greater than you want, start the script process. Otherwise, exit 0 if [ $loadLevel -gt $levelToCheck ]; then echo "" > $tmpfile echo "**************************************" >>$tmpfile echo "Date: $dt " >>$tmpfile echo "Check System Load & Processes " >>$tmpfile echo "**************************************" >>$tmpfile # Get more variables from system: httpdProcesses=`ps -def | grep httpd | grep -v grep | wc -l` # Show current load level: echo "Load Level Is: $loadLevel" >>$tmpfile echo "*************************************************" >>$tmpfile # Show number of httpd processes now running (not including children): echo "Number of httpd processes now: $httpdProcesses" >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Show process list: echo "Processes now running:" >>$tmpfile ps f -ef >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Show current MySQL info: echo "Results from mytop:" >>$tmpfile /usr/bin/mytop -u $dbusr -p $dbpw -b -d $db >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Show current top: echo "top now shows:" >>$tmpfile echo "top now shows:" >>$topfile /usr/bin/top -b -n1 >>$tmpfile /usr/bin/top -b -n1 >>$topfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Show current connections: echo "netstat now shows:" >>$tmpfile /bin/netstat -p >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Check disk space echo "disk space:" >>$tmpfile /bin/df -k >>$tmpfile echo "*************************************************" >>$tmpfile echo "" >>$tmpfile # Send results to log file: /bin/cat $tmpfile >>$logfile # And email results to sysadmin: /usr/bin/mutt -s "$machine has a high load level! - $dt" -a $mysqlLog -a $msgLog $mailstop <$tmpfile /usr/bin/mutt -s "$machine has a high load level! - $dt" $mailstop1 <$topfile echo "**************************************" >>$logfile # And then remove the temp file: rm $tmpfile rm $topfile fi
# exit 0

Replace Text in Plain Text Files from the Command Line

Jason Faulkner — Mon, 30 Aug 2010 10:00:50 +0000

A very useful function which is missing from the Windows library of command line tools is the ability to replace text in plain text files. A function like this can be used for a variety of practical tasks which many system admin’s perform, such as:

Update configuration/INI files to replace UNC paths.
Mass update user information stored in INI files on a Terminal/Citrix server.
Use in conjunction with scripts to deploy ‘templated’ data and then apply values to the copied files.

Our solution is a VBScript which interfaces with the Visual Basic Replace function. By placing this script into a location in your Windows PATH variable, you now have this functionality available at your disposal.

Uses

Once on your system, you can call the script by simply using the ReplaceText command. A few examples will illustrate ways you can use this:

Replace the word “null” with “n/a” in the C:\Data\Values.csv file:

ReplaceText “C:\Data\Values.csv” null n/a

Scan all INI files in the C:\Users (+ sub directories) folder replacing all occurrences of “Server=Old” with “Server=New” using a case insensitive search:

FORFILES /P “C:\Users” /M *.ini /S /C “Cmd /C ReplaceText @path Server=Old Server=New /I”

Scan all CFG files in the current user’s profile replacing “p@ssw0rd” with “PA$$woRd” using a case sensitive search:

FORFILES /P “%UserProfile%” /M *.cfg /S /C “Cmd /C ReplaceText @path p@ssw0rd PA$$woRd”

As you can see below, the script is very simple and can easily be modified to accommodate any special situations you may have. Alternately, you may want to create copies of the script which hardcode particular values so you can execute the command with a double-click and/or allow you to easily distribute it to others.

The Script

'Replace Text 'Written by: Jason Faulkner 'SysadminGeek.com

'This script should be placed in a folder specified in your system's PATH variable.

'Usage (WScript): 'ReplaceText FileName OldText NewText [/I]

' /I (optional) - Text matching is not case sensitive

Set oArgs = WScript.Arguments

intCaseSensitive = 0 For i = 3 to oArgs.Count-1 If UCase(oArgs(i)) = "/I" Then intCaseSensitive = 1 Next

Set oFSO = CreateObject("Scripting.FileSystemObject")

If Not oFSO.FileExists(oArgs(0)) Then WScript.Echo "Specified file does not exist." Else Set oFile = oFSO.OpenTextFile(oArgs(0), 1) strText = oFile.ReadAll oFile.Close

strText = Replace(strText, oArgs(1), oArgs(2), 1, -1, intCaseSensitive)

Set oFile = oFSO.OpenTextFile(oArgs(0), 2) oFile.WriteLine strText oFile.Close End If

Additional Notes

By default, Windows uses WScript to execute VBScript (VBS) files. The only problem this can cause is any errors and/or messages from the script will appear as popup boxes. For a command line tool, it is best these messages be displayed in the console. There are a couple of ways you can accomplish this.

Change the default handler of VBScript files to CScript by running this command from command prompt (with Administrator rights):

CScript //H:CScript

Run the ReplaceText script explicitly using the CScript command:

CScript “C:\Path\To\ReplaceText.vbs” //B FileName OldText NewText [/I]

As a special case, executing ReplaceText from a batch script typically implies CScript as the engine used regardless of the default handler. You will definitely want to test this though prior to relying on this functionality.

Download ReplaceText Script from SysadminGeek.com

Batch Script to Conditionally Restart an Application

Jason Faulkner — Fri, 27 Aug 2010 10:00:35 +0000

Common system and/or environmental events such as resuming from standby or losing network connection can cause problems for certain applications which expect to be always on and connected. So if you have a certain application which crashes or goes into “not responding” mode somewhat frequently and a restart is the only fix for it, we have a simple fix for you in the form of a customizable batch script to simply kill the application and restart it.

In addition to the obvious situation above, this script can be used for a variety of useful tasks, such as:

Easily restart an application by double-clicking or using a hot key.
Restart a program only when it is hung or not responding.
Run as a scheduled task to make sure an application is always running.
Anywhere else you want to automate conditional restarting of an application.

Customizing the script should be pretty self explanatory by the comments, so just configure the script appropriately and you are all set.

The Script

@ECHO OFF
ECHO Restart Application
ECHO Written by: Jason Faulkner
ECHO SysadminGeek.com
ECHO.
ECHO.

SETLOCAL EnableExtensions

REM Enter the application information.
SET AppName=Application Name
SET ExeFile=FileToLaunch.exe
SET ExePath=C:\Path\To\Application

REM Select the conditions to kill the application.
REM A value of 1 = Yes, 0 = No
SET KillIfRunning=1
SET KillIfNotResponding=1
SET KillIfUnknownStatus=1

REM Specify when to start the application:
REM 1 = Start only if the process was previous killed.
REM 0 = Start the application regardless.
SET StartOnlyIfKilled=1

SET KillStatus="%TEMP%\KillStatus.tmp.txt"
SET Success=0

ECHO Killing existing %AppName% instance...
IF {%KillIfRunning%}=={1} CALL :CheckKillStatus "%ExeFile%" "RUNNING"
IF {%KillIfNotResponding%}=={1} CALL :CheckKillStatus "%ExeFile%" "NOT RESPONDING"
IF {%KillIfUnknownStatus%}=={1} CALL :CheckKillStatus "%ExeFile%" "UNKNOWN"
ECHO.

IF {%StartOnlyIfKilled%}=={1} (
	IF {%Success%}=={0} GOTO End
)
ECHO Restarting %AppName%...
START "%ExeFile%" "%ExePath%\%ExeFile%"
ECHO.

IF EXIST %KillStatus% DEL /F /Q %KillStatus%

ENDLOCAL

:CheckKillStatus
ECHO Killing with status: %~2
TASKKILL /FI "STATUS eq %~2" /IM "%~1" /F > %KillStatus%
SET /P KillResult= < %KillStatus%
FOR /F "tokens=1,* delims=:" %%A IN ("%KillResult%") DO (
	ECHO %%A:%%B
	IF /I {%%A}=={SUCCESS} SET /A Success=%Success%+1
)

:End

Conclusion

While you can find applications out there which perform basically the same function, using a simple script such as this avoids having “yet another program” running in the background.

How to Export Your Windows Server’s DNS Records to a Web Page

Jason Faulkner — Mon, 23 Aug 2010 10:00:58 +0000

If you run a Windows Server which takes advantage of the built in DNS Server, you have a nice graphical interface for viewing and managing your DNS records. However, the vast majority of the time you probably just look at these records as opposed to updating them. This process is not difficult, but can be a hassle as you have to connect to the DNS Server machine through remote desktop, open DNS controls and locate the record. Wouldn’t it be easier if you could simply see this information over the web?

To make this functionality possible, we have a very simple script which exports your current DNS Server records to text files and makes them available via a simple indexed HTML file which can be accessed from any device with a web browser.

Configuration

Overall, the script’s configuration options are very straight forward. You simply need to configure the output location where you would like the destination files to end up. This folder will be populated with a ‘default.htm’ and ‘[domain].dns.zone.txt’ files. These names can be customized in the script as needed.

The script makes the assumption that you have named your DNS files using the default naming convention the Windows DNS Server uses ([domain].dns). If you are not using the default naming convention, the script will not work properly.

As an additional function, the script can delete unused DNS record files which are no longer active in your DNS Server. If enabled (off by default), when the export procedure fails for a DNS record file, meaning the domain was not found in the DNS Server, it is deleted. These unlinked DNS record files do not do any harm or consume any resources, so it is safe to leave them alone.

If you update your DNS records often, you can configure the script to run regularly through a scheduled task so you know the information you are viewing is always current. The output of the script is read-only so any changes made to the resulting files will not be reflected in your DNS Server.

How it Works

The script simply reads your current DNS files from the default Windows location and then interfaces with the DNSCmd command line tool to produce the output files. The DNSCmd tool is included with Server 2008, but Server 2003 machines must install the Resource Kit Tools to put this utility on your system.

The ‘[domain].dns.zone.txt’ are the output produced by the ZoneExport command.

You can access the listing by viewing the output ‘default.htm’ file in a browser. If you have configured the script to export to a publically available location, you can view the output from anywhere.

By clicking on a domain, you can see all the DNS information from your DNS Server for that domain.

The Script

@ECHO OFF TITLE DNS Dump to HTML ECHO DNS Dump to HTML ECHO Written by: Jason Faulkner ECHO SysadminGeek.com ECHO. ECHO.

SETLOCAL EnableDelayedExpansion

REM Directory where the HTML pages should be generated. SET OutPath=C:\inetpub\wwwroot\dns SET HTMLPage=default.htm

REM HTML page title/header. SET Title=DNS Records

REM Delete DNS record files which are not currently loaded in the DNS server (1=Yes, 0=No) SET DeleteNotFound=0

DEL /Q "%OutPath%\*" SET OutFile="%OutPath%\%HTMLPage%"

REM HTML header info. Customize as needed. ECHO ^<HTML^> >> %OutFile% ECHO ^<HEAD^> >> %OutFile% ECHO ^<TITLE^>%Title%^</TITLE^> >> %OutFile% ECHO ^</HEAD^> >> %OutFile% ECHO ^<BODY^> >> %OutFile% ECHO ^<H1^>%Title%^<H1^> >> %OutFile% ECHO ^<H3^>Machine Name: %ComputerName%^</H3^> >> %OutFile% ECHO ^<H5^>Generated on: %Date% %Time%^</H5^> >> %OutFile%

SET DNSDir=%WinDir%\system32\dns FOR /F %%A IN ('DIR /A:-D /B /L %DNSDir%\*.dns') DO ( SET Zone=%%A SET Zone=!Zone:.dns=! SET ZoneFile=!Zone!.dns.zone.txt ECHO Exporting: !Zone! DNSCmd . /ZoneExport !Zone! !ZoneFile! IF NOT EXIST %DNSDir%\!ZoneFile! ( ECHO !Zone! is not currently loaded in DNS Server. IF {%DeleteNotFound%}=={1} DEL /F /Q %DNSDir%\%%A ) ELSE ( ECHO ^<A HREF="!ZoneFile!"^>!Zone!^</A^>^<BR/^> >> %OutFile% REM Output is always to DNS directory, so move the file to the HTML dir. MOVE /Y %DNSDir%\!ZoneFile! "%OutPath%\!ZoneFile!" ) ECHO. )

ECHO ^<BR/^> >> %OutFile% ECHO ^</BODY^> >> %OutFile% ECHO ^</HTML^> >> %OutFile%

ENDLOCAL

Download DNS Dump to HTML Script from SysadminGeek.com

Download Windows Server 2003 Resource Kit Tools from Microsoft