Tuesday, September 30, 2008
SELinux - dealing with exceptions
So we're seeing errors in our /var/log/messages like:

Sep 28 03:52:51 fvs-pri setroubleshoot: SELinux is preventing freshclam (freshclam_t) "read" to ./main.cld (var_t). For complete SEL
inux messages. run sealert -l 10ce7bfb-6c44-473e-94a1-4691c04d2bef
Sep 28 03:52:51 fvs-pri setroubleshoot: SELinux is preventing freshclam (freshclam_t) "write" to ./clamav (var_t). For complete SELi
nux messages. run sealert -l 276efeb4-6990-497f-bcf0-6df0327c6f52

It's fairly easy to write exceptions, using audit2allow.

For example:

# cd /usr/share/selinux/devel/

# egrep "(clam)" /var/log/audit/audit.log /var/log/audit/audit.log.1 | audit2allow -M clam20081230

# /usr/sbin/semodule -i clam20080930.pp

Note: This is the very quick and dirty way of dealing with exceptions - it really doesn't fix the underlying issue.

Labels:

Friday, August 08, 2008
Debugging SSL connections
We're experiencing odd delays when talking to our mail server over SMTPS (SSL). I just found this post which helps us debug it.

How to debug SSL SMTP - by Sébastien Wains

$ openssl s_client -connect mail.example.com:465

Labels:

Wednesday, August 06, 2008
Linux RAID tuning and troubleshooting
Ran across this while searching another topic.

http://makarevitch.org/rant/raid/

Labels:

Friday, August 01, 2008
ngg.js and fgg.js site infections
One of our users visited a website that was infected with the ngg.js and fgg.js codes (they get injected into the HTML files on the server towards the end of the page).

We've blocked it in our squid configuration by:

# squid.conf

acl blocked_urls dstdomain "/etc/squid/blocked_urls.squid"
acl blocked_regex urlpath_regex "/etc/squid/blocked_regex.squid"

# Block some URLs
http_access deny blocked_urls
http_access deny blocked_regex

# blocked_urls.squid
.bjxt.ru
.njep.ru
.uhwc.ru

# blocked_regexp.squid
/fgg\.js
/ngg\.js

I won't explain this too much except to say that the blocked_urls file is designed to block top-level domains, while the regexp file is for blocking URLs using a regular expression.

Labels: ,

Monday, July 28, 2008
Dovecot - CMUSieve Errors
After upgrading our CentOS 5 box to the latest revisions this week (including Dovecot 1.1), we're seeing the following error message in the log files. Sieve was working fine with Dovecot 1.0.

# cat /var/vmail/dovecot-deliver.log

deliver(ruth@example.com): Jul 28 11:11:44 Error: dlopen(/usr/lib64/dovecot/lda/lib90_cmusieve_plugin.so) failed: /usr/lib64/dovecot/lda/lib90_cmusieve_plugin.so: undefined symbol: message_decoder_init
deliver(ruth@example.com): Jul 28 11:11:44 Fatal: Couldn't load required plugins

# ls -l /usr/libexec/dovecot/sievec
-rwxr-xr-x 1 root root 165152 Jun 11 03:21 /usr/libexec/dovecot/sievec

# yum list | grep "dovecot"
dovecot.x86_64 1:1.1.1-2_76.el5 installed
dovecot-sieve.x86_64 1.1.5-8.el5 installed
dovecot.x86_64 1:1.1.2-2_77.el5 atrpms
dovecot-devel.x86_64 1:1.1.2-2_77.el5 atrpms

Not sure yet what went wrong during the upgrade.

...

Update: The problem was that we had made a copy of Dovecot's "deliver" executable to make it setuid to work with virtual user local delivery. After the update, we forgot to update this copy of the exectuable.

Once we updated the setuid copy of "deliver", things worked fine.

Labels: ,

Monday, July 14, 2008
fsvs urls or fsvs initialize results in No such file or directory (2) error
So I was setting up FSVS 1.1.16 on a new CentOS 5.1 box this week (one of the first things that I do as soon as possible before configuration starts). And I encountered the following error:

# fsvs -v urls svn+ssh://svn.example.com/sys-machinename

An error occurred at 14:40:31.865: No such file or directory (2)
in url__output_list
in url__work
in main: action urls failed

...

The fix is to create the "/etc/fsvs" folder

fsvs 1.1.16 was smart enough to remind me to create /var/spool/fsvs, but it apparently doesn't give a good error message when the "/etc/fsvs" folder does not exist.

Labels:

Saturday, June 28, 2008
FSVS - Install on CentOS 5
The following should be enough (and is probably overkill) to install all of the dependencies that FSVS 1.1.16 needs on CentOS 5 (and CentOS 5.1)

# yum install subversion subversion-devel ctags apr apr-devel gcc gdbm gdbm-devel pcre pcre-devel apr-util-devel

# ./configure
configure: *** Now configuring FSVS ***
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
configure: "CFLAGS=-g -O2 -D_GNU_SOURCE=1 -D_FILE_OFFSET_BITS=64 -idirafter /usr/local/include -idirafter /usr/include -idirafter /openpkg/include -idirafter /usr/include/apr-1"
configure: "LDFLAGS= -L/usr/local/lib -L/openpkg/lib"
checking for pcre_compile in -lpcre... yes
checking for apr_md5_init in -laprutil-1... no
configure: error: Sorry, can't find APR.
See `config.log' for more details.

Note the addition of "apr-util-devel" at the end of the "yum install" line. This fixes the error when you run ./configure for FSVS and get the "can't find APR" error.

In older versions of CentOS 5, we did not need to also specify the apr-util-devel package.

Labels:

Monday, March 10, 2008
Postgresql 8.1 under a nondefault directory with SELinux
So I like to keep my PostgreSQL install in a non-standard location. Normally, this is as easy as setting PGDATA= in the /etc/sysconfig/pgsql/postgresql file. But when SELinux is installed, you also have to deal with system context issues.

One symptom of this is that /etc/init.d/postgresql start will fail, but starting the database interactively using the "su postgres" and pg_ctl commands will work. This is because SELinux is a lot stricter with programs started in the startup scripts vs programs that are started from an interactive shell.

If you dig through the pgsql-general mail archives, you'll find a thread titled "[GENERAL] Using an alternate PGDATA on RHEL4 with SELinux enabled" from July 2006. Unfortunately, nobody posted the answer for how to work around this issue and the original poster merely disabled SELinux. Tom Lane in particular says:

The default selinux policy prevents postgres from writing anywhere except under /var/lib/pgsql. If you want a nondefault PGDATA location then you have to tweak the policy.

However, I've stumbled across this "Just Someone Re: SELinux + CREATE TABLESPACE = ?" which gives some insights into the issue. It was also posted to the pgsql-general mailing list, but about a week later. I quote "Just Someone":

If you rather keep SELinux on, you can still set the SELinux context
on the directory where you want the tablespaces to one postgres will
like.

To find what is the permissions you need, you can use ls -Z. It will
list the SELinux context. Check /var/lib/pgsql/data (or wherever
postgres data is pointing to), and then set this same permission on
the target dir using chcon.

For example, on my FC4 system all subdirectories on the data directory have:
root:object_r:postgresql_db_t or user_u:object_r:postgresql_db_t

So if you want to chage /path/to/foo/which/is/not/under/pgdata, run
(as root or sudo):

chcon root:object_r:postgresql_db_t /path/to/foo/which/is/not/under/pgdata

This way postgres can access it, and you get the SELinux security.

Bye,

Guy.


So basically, we need to look at the context of the existing /var/lib/pgsql folder and then make our new directories to match that. We'll start by looking at /var/lib/pgsql:

# ls -Z /var/lib/
drwx------ postgres postgres system_u:object_r:var_lib_t pgsql


Let's compare this to our new location:

# ls -Z /var/
drwxr-xr-x root root system_u:object_r:file_t pgsql


Yeah, that's definitely not correct. So let's fix it:

# chown postgres:postgres /var/pgsql
# chmod 700 /var/pgsql
# chcon system_u:object_r:var_t /var/pgsql
# ls -Z /var/
drwx------ postgres postgres system_u:object_r:var_lib_t pgsql


Which now matches exactly what we saw for /var/lib/pgsql. Now we need to do the same thing for the contents of /var/pgsql as compared to /var/lib/pgsql.

# ls -Z /var/lib/pgsql
drwx------ postgres postgres system_u:object_r:var_lib_t backups
drwx------ postgres postgres system_u:object_r:postgresql_db_t data
-rw------- postgres postgres system_u:object_r:postgresql_log_t pgstartup.log


As compared to:

# ls -Z /var/pgsql
drwx------ postgres postgres user_u:object_r:var_log_t data
drwx------ root root system_u:object_r:file_t lost+found


Once again, things need to be fixed up.

# su postgres
$ mkdir /var/pgsql/backups
$ chmod 700 /var/pgsql/backups
$ chcon system_u:object_r:var_t /var/pgsql/backups
$ chcon system_u:object_r:postgresql_db_t /var/pgsql/data
$ touch /var/pgsql/pgstartup.log
$ chmod 600 /var/pgsql/pgstartup.log
$ chcon system_u:object_r:postgresql_log_t /var/pgsql/pgstartup.log
$ ls -Z /var/pgsql
drwx------ postgres postgres system_u:object_r:var_t backups
drwx------ postgres postgres system_u:object_r:postgresql_db_t data
drwx------ root root system_u:object_r:file_t lost+found
-rw------- postgres postgres system_u:object_r:postgresql_log_t pgstartup.log
$ ls -Z /var/lib/pgsql
drwx------ postgres postgres system_u:object_r:var_lib_t backups
drwx------ postgres postgres system_u:object_r:postgresql_db_t data
-rw------- postgres postgres system_u:object_r:postgresql_log_t pgstartup.log


Which looks correct. At least our ownership, file attributes and file context all match the original. Note that I left the context of some things as system_u:object_r:var_t instead of system_u:object_r:var_lib_t.

Now for the hard part, we have to look at ALL of the subdirectory contents under /var/lib/pgsql and match them up in the new location:

$ cd /var/lib/pgsql ; ls -RZ
.:
drwx------ postgres postgres system_u:object_r:var_lib_t backups
drwx------ postgres postgres system_u:object_r:postgresql_db_t data
-rw------- postgres postgres system_u:object_r:postgresql_log_t pgstartup.log

./backups:

./data:
drwx------ postgres postgres user_u:object_r:postgresql_db_t base
drwx------ postgres postgres user_u:object_r:postgresql_db_t global
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_clog
-rw------- postgres postgres user_u:object_r:postgresql_db_t pg_hba.conf
-rw------- postgres postgres user_u:object_r:postgresql_db_t pg_ident.conf
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_log
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_multixact
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_subtrans
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_tblspc
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_twophase
-rw------- postgres postgres user_u:object_r:postgresql_db_t PG_VERSION
drwx------ postgres postgres user_u:object_r:postgresql_db_t pg_xlog
-rw------- postgres postgres user_u:object_r:postgresql_db_t postgresql.conf
-rw------- postgres postgres system_u:object_r:postgresql_db_t postmaster.opts

./data/base:
drwx------ postgres postgres user_u:object_r:postgresql_db_t 1
drwx------ postgres postgres user_u:object_r:postgresql_db_t 10792
drwx------ postgres postgres user_u:object_r:postgresql_db_t 10793

./data/base/1:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10287
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10289
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10293
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10295
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10299
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10301
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10302
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10304
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10305
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10307
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10308
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10310
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10723
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10725
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10727
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10728
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10730
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10732
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10733
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10735
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10737
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10738
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10740
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10742
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10743
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10745
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10747
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10748
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10750
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10752
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1247
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1248
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1249
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1250
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1255
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1259
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2600
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2601
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2602
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2603
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2604
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2605
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2606
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2607
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2608
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2609
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2610
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2611
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2612
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2613
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2614
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2615
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2616
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2617
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2618
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2619
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2620
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2650
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2651
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2652
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2653
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2654
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2655
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2656
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2657
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2658
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2659
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2660
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2661
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2662
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2663
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2664
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2665
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2666
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2667
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2668
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2669
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2670
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2673
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2674
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2675
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2678
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2679
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2680
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2681
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2682
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2683
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2684
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2685
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2686
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2687
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2688
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2689
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2690
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2691
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2692
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2693
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2696
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2699
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2700
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2701
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2702
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2703
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2704
-rw------- postgres postgres user_u:object_r:postgresql_db_t PG_VERSION

./data/base/10792:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10287
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10289
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10293
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10295
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10299
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10301
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10302
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10304
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10305
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10307
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10308
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10310
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10723
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10725
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10727
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10728
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10730
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10732
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10733
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10735
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10737
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10738
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10740
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10742
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10743
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10745
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10747
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10748
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10750
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10752
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1247
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1248
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1249
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1250
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1255
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1259
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2600
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2601
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2602
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2603
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2604
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2605
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2606
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2607
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2608
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2609
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2610
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2611
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2612
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2613
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2614
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2615
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2616
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2617
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2618
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2619
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2620
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2650
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2651
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2652
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2653
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2654
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2655
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2656
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2657
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2658
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2659
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2660
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2661
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2662
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2663
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2664
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2665
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2666
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2667
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2668
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2669
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2670
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2673
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2674
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2675
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2678
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2679
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2680
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2681
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2682
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2683
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2684
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2685
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2686
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2687
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2688
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2689
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2690
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2691
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2692
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2693
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2696
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2699
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2700
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2701
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2702
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2703
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2704
-rw------- postgres postgres user_u:object_r:postgresql_db_t PG_VERSION

./data/base/10793:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10287
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10289
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10293
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10295
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10299
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10301
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10302
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10304
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10305
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10307
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10308
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10310
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10723
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10725
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10727
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10728
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10730
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10732
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10733
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10735
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10737
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10738
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10740
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10742
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10743
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10745
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10747
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10748
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10750
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10752
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1247
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1248
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1249
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1250
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1255
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1259
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2600
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2601
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2602
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2603
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2604
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2605
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2606
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2607
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2608
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2609
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2610
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2611
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2612
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2613
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2614
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2615
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2616
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2617
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2618
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2619
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2620
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2650
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2651
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2652
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2653
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2654
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2655
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2656
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2657
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2658
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2659
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2660
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2661
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2662
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2663
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2664
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2665
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2666
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2667
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2668
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2669
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2670
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2673
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2674
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2675
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2678
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2679
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2680
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2681
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2682
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2683
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2684
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2685
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2686
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2687
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2688
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2689
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2690
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2691
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2692
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2693
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2696
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2699
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2700
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2701
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2702
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2703
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2704
-rw------- postgres postgres system_u:object_r:postgresql_db_t pg_internal.init
-rw------- postgres postgres user_u:object_r:postgresql_db_t PG_VERSION

./data/global:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10290
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10292
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10296
-rw------- postgres postgres user_u:object_r:postgresql_db_t 10298
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1136
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1137
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1213
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1214
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1232
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1233
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1260
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1261
-rw------- postgres postgres user_u:object_r:postgresql_db_t 1262
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2671
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2672
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2676
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2677
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2694
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2695
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2697
-rw------- postgres postgres user_u:object_r:postgresql_db_t 2698
-rw------- postgres postgres system_u:object_r:postgresql_db_t pg_auth
-rw------- postgres postgres user_u:object_r:postgresql_db_t pg_control
-rw------- postgres postgres system_u:object_r:postgresql_db_t pg_database
-rw------- postgres postgres system_u:object_r:postgresql_db_t pg_fsm.cache
-rw------- postgres postgres system_u:object_r:postgresql_db_t pgstat.stat

./data/pg_clog:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 0000

./data/pg_log:
-rw------- postgres postgres system_u:object_r:postgresql_db_t postgresql-Mon.log
-rw------- postgres postgres system_u:object_r:postgresql_db_t postgresql-Sat.log
-rw------- postgres postgres system_u:object_r:postgresql_db_t postgresql-Sun.log
-rw------- postgres postgres system_u:object_r:postgresql_db_t postgresql-Tue.log

./data/pg_multixact:
drwx------ postgres postgres user_u:object_r:postgresql_db_t members
drwx------ postgres postgres user_u:object_r:postgresql_db_t offsets

./data/pg_multixact/members:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 0000

./data/pg_multixact/offsets:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 0000

./data/pg_subtrans:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 0000

./data/pg_tblspc:

./data/pg_twophase:

./data/pg_xlog:
-rw------- postgres postgres user_u:object_r:postgresql_db_t 000000010000000000000000
drwx------ postgres postgres user_u:object_r:postgresql_db_t archive_status

./data/pg_xlog/archive_status:
$


Keep that list open in a text editor, or something else because you'll need to refer to it frequently. We can fix most of it by making everything set to context "user_u:object_r:postgresql_db_t" to start. Which is a brute-force approach.

$ chcon -R user_u:object_r:postgresql_db_t *

Now we can go and start fixing things that should not be that particular context. Now, it's quite probable that this is overkill, but I believe in being thorough.

$ chcon system_u:object_r:postgresql_db_t postmaster.opts
$ find . -name pg_internal.init -exec chcon system_u:object_r:postgresql_db_t {} \;
$ chcon system_u:object_r:postgresql_db_t global/pg_auth
$ chcon system_u:object_r:postgresql_db_t global/pg_database
$ chcon system_u:object_r:postgresql_db_t global/pg_fsm.cache
(file may not exist)
$ chcon system_u:object_r:postgresql_db_t global/pgstat.stat
$ chcon system_u:object_r:postgresql_db_t pg_log/postgresql-*.log


At this point, your postgresql data directory SHOULD be configured correctly. (No guarantees!) So now you can restart postgresql (/etc/init.d/postgresql start) and it will work properly in the new location.

Notes:

- Tested on CentOS 5 (or CentOS 5.1), it should also work on RedHat Linux.

- If you ever re-tag the entire filesystem with SELinux, you will (probably) have to go back and re-tag your postgresql data directory.

- Because of the above note, it may be better to mount the LVM or SAN partition for PostgreSQL at the default location of /var/lib/pgsql instead of forcing it into another location. On the other hand, as long as you know how to fix it and don't re-tag indiscriminately, SELinux should never get in the way again.

Labels: ,

Thursday, January 03, 2008
Subversion repository creation cheatsheet
Whenever I setup new SVN repositories, I always create a unix group for people who need read/write access to the repository. If the repository is named "tgh-public", then I choose to name the group as "svn-tgh-public".

I also generally designate a single user as the initial owner of the SVN repository folder under /var/svn. Alternately, you could just leave the repository owned by root.

# cd /var/svn
(your repositories may be stored elsewhere)
# /usr/sbin/groupadd svn-repositoryname
# svnadmin create /var/svn/repositoryname
# chmod -R 770 repositoryname
# chmod -R g+s repositoryname/db
# chown -R username:svn-repositoryname repositoryname
# /usr/sbin/usermod -a -G svn-repositoryname username

Notes:

  • You'll want to repeat the "usermod" command for each person who will have access to the new repository.
  • The chmod value of 770 means that anyone who is either the "username" or who belongs to the "svn-repositoryname" group will be able to access this repository via SVN+SSH and make changes.
  • If you want to allow public reads, then you should use a chmod value of 774 which allows everyone read access to the folders.
  • Forgetting to set the sticky bit for the group will result in a repository that breaks as different users edit the contents. If you dig through the contents of the FSFS directories, you'll see that files were created with ownership username:username instead of belonging to the group who is responsible for that repository.

Labels:

Friday, December 21, 2007
Replacing a failed drive in a Software RAID mirror set
Like I wrote about last time, I have a failing drive in my triple active RAID mirror set on my firewall box. See also "Failing hard drive in a Software RAID". I'm still trying to decide whether the disk has actually failed, or if it is just having issues.

# /sbin/badblocks -sv /dev/sdc2

Since I have unmounted this RAID slice, I'm going to test with a DESTRUCTIVE write/read verification. (Which is also a good way to wipe the disk.)

# /sbin/badblocks -sv -w -t random /dev/sdc2

Well, after a few runs with that, the disk is no longer making "retry" noises. So I'm going to re-add the slice to the RAID array and see what happens.

# /sbin/mdadm /dev/md1 -a /dev/sdc2

And force mdadm to verify the sync:

# echo check > /sys/block/md1/md/sync_action

It seems to be working. I'm guessing that I finally convinced SMART to re-map the bad sector that was causing problems.

Labels:

Wednesday, December 05, 2007
Failed drive slice in a Software RAID after resync
One of the things that I do periodically on my servers is to run a mdadm resync. Because this can put a heavy strain on the disk system, I strongly suggest that you have good backups in place. My home systems run a check about once a month, servers at work run a check early on Tuesday mornings.

The script is very simple, and you can even fire off the command by writing "check" to the sync_action variable of the md process.

#!/bin/sh
# Tells mdadm to verify that the arrays are synchronized.
# This deals with the issue where a seldom-read disk block has gone bad
# by doing a daily/weekly verification of the array.

echo check > /sys/block/md0/md/sync_action
echo check > /sys/block/md1/md/sync_action
echo check > /sys/block/md2/md/sync_action
echo check > /sys/block/md3/md/sync_action
echo check > /sys/block/md4/md/sync_action
echo check > /sys/block/md5/md/sync_action
echo check > /sys/block/md6/md/sync_action


In this particular case, all of my RAID slices verified correctly, except for one of them. In this particular situation I'm running a triple-active RAID1 array. (Instead of using a hot-spare disk, I'm putting live data onto all three disks and using all three actively.)

See also Failing hard drive in a Software RAID

$ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[2] sdb1[1] sda1[0]
256896 blocks [3/3] [UUU]

md2 : active raid1 sdc3[2] sdb3[1] sda3[0]
12289600 blocks [3/3] [UUU]

md4 : active raid1 sdc5[2] sdb5[1] sda5[0]
33551616 blocks [3/3] [UUU]

md3 : active raid1 sdc6[2] sdb6[1] sda6[0]
1052160 blocks [3/3] [UUU]

md5 : active raid1 sdc7[2] sdb7[1] sda7[0]
64010880 blocks [3/3] [UUU]

md6 : active raid1 sdc8[2] sdb8[1] sda8[0]
267257216 blocks [3/3] [UUU]

md7 : active raid1 sdf1[2] sde1[1] sdd1[0]
488383936 blocks [3/3] [UUU]

md1 : active raid1 sdc2[3](F) sdb2[1] sda2[0]
12289600 blocks [3/2] [UU_]

unused devices: <none>


The md1 array is my / (root) partition. Since the rest of the disk slices appear to be fine, I'm going to proceed with the assumption that it was a minor glitch.

Step 0: Analyze the failure

The first sign of error was the (F) showing up in /proc/mdstat. Apparently I don't have mdadm configured yet in monitor mode so that it e-mails me when it finds an error.

# grep "sdc2" messages
Dec 4 09:11:58 fw1-shimo kernel: raid1: Disk failure on sdc2, disabling device.
Dec 4 09:12:06 fw1-shimo kernel: disk 2, wo:1, o:0, dev:sdc2


The full detail from the mdadm resync:

# grep "Dec 4 09" messages | grep "md:"
Dec 4 09:08:33 fw1-shimo kernel: md: md6: sync done.
Dec 4 09:08:33 fw1-shimo kernel: md: syncing RAID array md1
Dec 4 09:08:33 fw1-shimo kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Dec 4 09:08:33 fw1-shimo kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
Dec 4 09:08:33 fw1-shimo kernel: md: using 128k window, over a total of 12289600 blocks.
Dec 4 09:11:31 fw1-shimo kernel: md: md1: sync done.
#


And finally, evidence from the logs that shows that sdc was having issues:

Dec 4 09:11:34 fw1-shimo kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:34 fw1-shimo kernel: ata3.00: (BMDMA stat 0x60)
Dec 4 09:11:34 fw1-shimo kernel: ata3.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:34 fw1-shimo kernel: ata3: EH complete
Dec 4 09:11:35 fw1-shimo kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:35 fw1-shimo kernel: ata2.00: (BMDMA stat 0x0)
Dec 4 09:11:35 fw1-shimo kernel: ata2.00: tag 0 cmd 0xc8 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:35 fw1-shimo kernel: ata2: EH complete
Dec 4 09:11:37 fw1-shimo kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:37 fw1-shimo kernel: ata3.00: (BMDMA stat 0x60)
Dec 4 09:11:37 fw1-shimo kernel: ata3.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:37 fw1-shimo kernel: ata3: EH complete
Dec 4 09:11:50 fw1-shimo kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:51 fw1-shimo kernel: ata3.00: (BMDMA stat 0x60)
Dec 4 09:11:51 fw1-shimo kernel: ata3.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:51 fw1-shimo kernel: ata3: EH complete
Dec 4 09:11:51 fw1-shimo kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:52 fw1-shimo kernel: ata3.00: (BMDMA stat 0x60)
Dec 4 09:11:52 fw1-shimo kernel: ata3.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:52 fw1-shimo kernel: ata3: EH complete
Dec 4 09:11:52 fw1-shimo setroubleshoot: SELinux is preventing /usr/sbin/sendmail.postfix (system_mail_t) "read" to /dev/md1 (proc_mdstat_t). For complete SELinux messages. run sealert -l d5c655f4-6fc3-445b-ab9d-3b21336cb2d0
Dec 4 09:11:52 fw1-shimo kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:53 fw1-shimo kernel: ata3.00: (BMDMA stat 0x60)
Dec 4 09:11:53 fw1-shimo kernel: ata3.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:53 fw1-shimo kernel: ata3: EH complete
Dec 4 09:11:53 fw1-shimo kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Dec 4 09:11:53 fw1-shimo kernel: ata3.00: (BMDMA stat 0x60)
Dec 4 09:11:54 fw1-shimo kernel: ata3.00: tag 0 cmd 0x25 Emask 0x9 stat 0x51 err 0x40 (media error)
Dec 4 09:11:54 fw1-shimo kernel: sd 2:0:0:0: SCSI error: return code = 0x08000002
Dec 4 09:11:54 fw1-shimo kernel: sdc: Current: sense key: Medium Error
Dec 4 09:11:54 fw1-shimo kernel: Additional sense: Unrecovered read error - auto reallocate failed
Dec 4 09:11:55 fw1-shimo kernel: end_request: I/O error, dev sdc, sector 25091744
Dec 4 09:11:55 fw1-shimo kernel: ata3: EH complete
Dec 4 09:11:55 fw1-shimo kernel: SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
Dec 4 09:11:55 fw1-shimo kernel: sdc: Write Protect is off
Dec 4 09:11:56 fw1-shimo kernel: SCSI device sdc: drive cache: write back
Dec 4 09:11:56 fw1-shimo kernel: SCSI device sdb: 781422768 512-byte hdwr sectors (400088 MB)
Dec 4 09:11:56 fw1-shimo kernel: sdb: Write Protect is off
Dec 4 09:11:57 fw1-shimo kernel: SCSI device sdb: drive cache: write back
Dec 4 09:11:57 fw1-shimo kernel: SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
Dec 4 09:11:57 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:11:57 fw1-shimo kernel: counted 127, received 15
Dec 4 09:11:58 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:11:58 fw1-shimo kernel: raid1: Disk failure on sdc2, disabling device.
Dec 4 09:11:58 fw1-shimo kernel: Operation continuing on 2 devices
Dec 4 09:11:58 fw1-shimo kernel: blk: request botched
Dec 4 09:11:58 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:11:59 fw1-shimo kernel: counted 112, received 16
Dec 4 09:11:59 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:11:59 fw1-shimo kernel: blk: request botched
Dec 4 09:11:59 fw1-shimo kernel: sdc: Write Protect is off
Dec 4 09:12:00 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:12:00 fw1-shimo kernel: counted 96, received 16
Dec 4 09:12:00 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:12:01 fw1-shimo kernel: blk: request botched
Dec 4 09:12:01 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:12:01 fw1-shimo kernel: counted 80, received 16
Dec 4 09:12:01 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:12:02 fw1-shimo kernel: blk: request botched
Dec 4 09:12:02 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:12:02 fw1-shimo kernel: counted 64, received 16
Dec 4 09:12:02 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:12:03 fw1-shimo kernel: blk: request botched
Dec 4 09:12:03 fw1-shimo kernel: SCSI device sdc: drive cache: write back
Dec 4 09:12:03 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:12:03 fw1-shimo kernel: counted 48, received 16
Dec 4 09:12:04 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:12:04 fw1-shimo kernel: blk: request botched
Dec 4 09:12:04 fw1-shimo kernel: Incorrect number of segments after building list
Dec 4 09:12:04 fw1-shimo kernel: counted 32, received 16
Dec 4 09:12:05 fw1-shimo kernel: req nr_sec 0, cur_nr_sec 8
Dec 4 09:12:05 fw1-shimo kernel: blk: request botched
Dec 4 09:12:05 fw1-shimo kernel: ata3.00: WARNING: zero len r/w req
Dec 4 09:12:06 fw1-shimo last message repeated 5 times


Step 1: Drop the failed slice

# /sbin/mdadm /dev/md1 --fail /dev/sdc2
mdadm: set /dev/sdc2 faulty in /dev/md1
# /sbin/mdadm /dev/md1 --remove /dev/sdc2
mdadm: hot removed /dev/sdc2


Step 2: Zero out the failed slice

My thinking here is that by zeroing out the failed slice, I can force the SATA disk to remap any sectors that have gone bad.

# dd if=/dev/zero of=/dev/sdc2
dd: writing to `/dev/sdc2': Input/output error
24577993+0 records in
24577992+0 records out
12583931904 bytes (13 GB) copied, 1916.7 seconds, 6.6 MB/s


Well, that's not a good sign (and the disk was clicking a bit). So I'll run smartctl and check the disk's SMART info (see Monitoring Hard Disks with SMART).

# /usr/sbin/smartctl -i -d ata /dev/sdc
smartctl version 5.36 [x86_64-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model: SAMSUNG HD400LJ
Serial Number: S0H2J1KLA07831
Firmware Version: ZZ100-15
User Capacity: 400,088,457,216 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 T13 1532D revision 4a
Local Time is: Wed Dec 5 09:43:36 2007 EST

==> WARNING: May need -F samsung or -F samsung2 enabled; see manual for details.

SMART support is: Available - device has SMART capability.
SMART support is: Enabled


However, the "-Hc" output of smartctl says that the disk health is still "PASSED" and not "FAILING". So it's possible that the disk doesn't need to be retired yet.

# /usr/sbin/smartctl -Hc -d ata /dev/sdc
smartctl version 5.36 [x86_64-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x05) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (7640) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 130) minutes.


Personally, since I know the drive makes clicking noises and throws an error during the dd wipe, I'm going to swap it out.

Labels:

Wednesday, August 15, 2007
Installing Angband on CentOS 5
Installation of Angband 3.0.9 on RedHat or CentOS5.

1) Grab the latest source release from http://rephial.org/release

# cd /root
# wget http://rephial.org/downloads/3.0/angband-3.0.9-src.tar.gz
# tar xzf angband-3.0.9-src.tar.gz

2) Compile the source code (the following is for running angband from the location where you unpacked the source, see Compiling for other options)

# cd angband-3.0.9
# ./configure
# make
# make install

3a) Errors: Make can't find "ncurses.h" (see also Compiling)

# make
CC main-gcu.c
main-gcu.c:63:22: error: ncurses.h: No such file or directory

Which indicates that you need to install the ncurses library. You can fix that by installing the "ncurses-devel" and re-running "./configure".

# yum install ncurses-devel
# ./configure

4) If you do a system install (making Angband available for all users on the system), make sure you add the users to the "games" group. Otherwise, when your users attempt to run Angband, they will get error messages about not being able to write to various files in the /usr/local/games/lib/angband folders.

# ./configure --with-setgid=games --with-libpath=/usr/local/games/lib/angband --bindir=/usr/local/games
# make
# make install

Labels: ,

Thursday, July 05, 2007
SELinux is preventing named from write access
It seems like the SELinux profile in CentOS5 may not be correct by default. In my /var/log/messages file, I have thousands of entries per month consisting of:

Jul 4 05:01:04 fw1-hosho setroubleshoot: SELinux is preventing /usr/sbin/named (named_t) "write" access to named (named_conf_t). For complete SELinux messages. run sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990

Here's the output of the sealert command:

# sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990
Summary
SELinux is preventing /usr/sbin/named (named_t) "write" access to named
(named_conf_t).

Detailed Description
SELinux denied access requested by /usr/sbin/named. It is not expected that
this access is required by /usr/sbin/named and this access may signal an
intrusion attempt. It is also possible that the specific version or
configuration of the application is causing it to require additional access.
Please file a http://bugzilla.redhat.com/bugzilla/enter_bug.cgi against this
package.

Allowing Access
Sometimes labeling problems can cause SELinux denials. You could try to
restore the default system file context for named, restorecon -v named.
There is currently no automatic way to allow this access. Instead, you can
generate a local policy module to allow this access - see
http://fedora.redhat.com/docs/selinux-faq-fc5/#id2961385 - or you can
disable SELinux protection entirely for the application. Disabling SELinux
protection is not recommended. Please file a
http://bugzilla.redhat.com/bugzilla/enter_bug.cgi against this package.
Changing the "named_disable_trans" boolean to true will disable SELinux
protection this application: "setsebool -P named_disable_trans=1."

The following command will allow this access:
setsebool -P named_disable_trans=1

Additional Information

Source Context system_u:system_r:named_t
Target Context root:object_r:named_conf_t
Target Objects named [ dir ]
Affected RPM Packages bind-9.3.3-8.el5 [application]
Policy RPM selinux-policy-2.4.6-30.el5
Selinux Enabled True
Policy Type targeted
MLS Enabled True
Enforcing Mode Enforcing
Plugin Name plugins.disable_trans
Host Name fw1-shimo.hq.example.org.
Platform Linux fw1-shimo.hq.example.org.
2.6.18-8.1.6.el5 #1 SMP Thu Jun 14 17:29:04 EDT
2007 x86_64 x86_64
Alert Count 70481
Line Numbers

Raw Audit Messages

avc: denied { write } for comm="named" dev=md1 egid=25 euid=25
exe="/usr/sbin/named" exit=-13 fsgid=25 fsuid=25 gid=25 items=0 name="named"
pid=2628 scontext=system_u:system_r:named_t:s0 sgid=25
subj=system_u:system_r:named_t:s0 suid=25 tclass=dir
tcontext=root:object_r:named_conf_t:s0 tty=(none) uid=25


The most helpful web page that I've found so far is the thread "Permissions Issue starting Bind 9.3.1". The gist seems to be that RedHat (and CentOS) are using a chroot bind installation in conjunction with an SELinux policy that expects the bind configuration files to be in a non-chroot setup. But there aren't very clear instructions there on fixing it.

Labels: , ,

Wednesday, July 04, 2007
Setting up svn+ssh on an alternate point for TortoiseSVN
This builds off a post to the TortoiseSVN user list: Specifying custom port for svn+ssh: a workaround

  1. Right-click on the Pageant icon in the system tray (I'm assuming that you're loading the SSH public key that you use for SVN into Pageant).
  2. Choose "New Session"
  3. Enter the hostname / IP address and SSH port that you'll be connecting to. If you're going to connect as "svn+ssh://thomas@svn.tgharold.com:2222", then this would be "svn.tgharold.com" and "2222".
  4. Go back to the "Session" tab and name the session as "svn.tgharold.com:2222".

Now you will be able to use both TortoiseSVN and the command-line version of SVN to talk to your repository over the alternate SSH port.

Why do this?

This is useful for cases where you want to put a SVN server on a publicly accessible IP address. What you will find is that if you leave SSH running on the default port, you will be inviting attacks on your SSH server. On the other hand, if you put the SSH server on an alternate port, you'll find that it gets attacked a lot less often (1-2 orders of magnitude difference would be likely).

Since mid-Oct of last year (around 8.5 months), we've logged 90,300 attack attempts against our SSH server. Usually they come in batches of attempting to guess accounts that normally exist or by attacking a list of common usernames. Since we don't allow root login, we don't allow password authentication, we only allow public key authentication and our SSH keys are limited to running "svnserve -t", we have yet to see a break-in attempt succeed.

Labels: , ,

Monday, July 02, 2007
LVM and SELinux
I was a bit perplexed... I had created a LV called /dev/vg/svn, had it mounted, was reading/writing data to it with no issues. But after I rebooted the CentOS5 server, I'm unable to mount the LV.

[root@localhost /]# /usr/sbin/pvscan
PV /dev/md6 VG vg lvm2 [144.78 GB / 59.78 GB free]
Total: 1 [144.78 GB] / in use: 1 [144.78 GB] / in no VG: 0 [0 ]
[root@localhost /]# /usr/sbin/vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg" using metadata type lvm2
[root@localhost /]# /usr/sbin/lvscan
No volume groups found
[root@localhost /]# /usr/sbin/lvdisplay
No volume groups found
[root@localhost /]# /usr/sbin/lvdisplay vg
--- Logical volume ---
LV Name /dev/vg/svn
VG Name vg
LV UUID taYjia-BWWs-IWG3-313k-VoC2-ghik-01mFCg
LV Write Access read/write
LV Status NOT available
LV Size 85.00 GB
Current LE 21760
Segments 1
Allocation inherit
Read ahead sectors 0

[root@localhost /]#


So lvdisplay knows that the LV is there, but only if I tell it to look at the VG named "vg".

...

Turns out that it's an SELinux issue. Because SELinux was blocking access to the /etc/lvm/.cache file, it was causing problems. Fixing it was as simple as:

# cd /etc/lvm
# /sbin/restorecon -v .cache
# /usr/sbin/lvscan
inactive '/dev/vg/svn' [85.00 GB] inherit

Labels: ,

Sunday, July 01, 2007
CentOS5: Moving /var/log to a separate volume
One thing I like to do is put /var/log on its own volume. That keeps the root volume from overflowing and also gets the log files out of the way. However, in CentOS5 (and probably RHEL5), SELinux is probably going to complain unless we tell it to "fixup" the new filesystem.

  1. Create the filesystem (I use ext3, so # /sbin/mke2fs -j /dev/mdX)
  2. Mount it at a temporary location: # mkdir /mnt/log ; mount /dev/mdX /mnt/log
  3. Copy the contents: # cp -a /var/log/* /mnt/log/
  4. It may be necessary to "fixup" the new volume: # cd /mnt/log ; /sbin/restorecon -R *
  5. Edit the /etc/fstab file to mount the new volume at /var/log
  6. Reboot

AFAIK, that's the extent of what's needed. Looking at the directory listings using "ls -lZ" seems to show the correct SELinux flags on the files between the two different directories.

Labels: , ,

Friday, June 22, 2007
FSVS ignore patterns (v1.1.5)
Here's a list of the current ignore patterns that I use on my CentOS5 box.

# fsvs ignore dump
./backup
./dev
./home
./lost+found
./media
./mnt
./proc/**
./root/.mozilla/firefox/**/Cache/**
./root/.thumbnails/**
./selinux
./sys
./tmp
./var/cache/**
./var/lock/**
./var/log/**
./var/named/chroot/proc
./var/run/**
./var/spool/**
./var/tmp/**

There are a few commands that I use to keep my sanity:

# fsvs dump ignore | sort > /root/fsvs-ignore.txt
# sort /root/fsvs-ignore.txt | fsvs ignore load

I find that keeping my ignore files in a .txt file under /root makes it easier to work with them. I'm able to edit the text file, load the ignore patterns into FSVS and see whether it does what it should. If it's wrong, I re-edit the text file and load them back into FSVS.

...

After mucking with a new box for a week, here's the set of ignore filters that I'm using on another CentOS5 box. On this particular box, I'm only versioning configuration data (/etc, /var/named).

[root@fw1-shimo /]# fsvs ignore dump | sort
./backup/
./bin/
./dev/
./home/
./lib/
./lib64/
./lost+found
./media/
./mnt/
./proc/
./root/
./sbin/
./selinux/
./srv/
./sys/
./tmp/
./usr/bin/
./usr/include/
./usr/kerberos/
./usr/lib/
./usr/lib64
./usr/libexec/
./usr/local/bin/
./usr/local/include/
./usr/local/lib/
./usr/local/libexec/
./usr/local/share/
./usr/local/src/
./usr/sbin/
./usr/share/
./usr/share/applications/
./usr/share/backgrounds/
./usr/share/dict/
./usr/share/doc/
./usr/share/i18n/
./usr/share/info/
./usr/share/locale/
./usr/share/man/
./usr/share/pixmaps/
./usr/share/X11/
./usr/share/zoneinfo/
./usr/src/
./usr/tmp/
./usr/X11R6/
./var/cache/
./var/lib/
./var/lock/
./var/log/
./var/named/chroot/dev/
./var/named/chroot/proc/
./var/named/chroot/var/run/
./var/run/
./var/spool/
./var/svn/
./var/tmp/
./var/www/
[root@fw1-shimo /]#

Labels: ,

Wednesday, June 20, 2007
Remote GUI administration of CentOS5 using Windows
Over the years, I've become very spoiled by Windows Terminal Services that we use to administer our Windows 2000 and Windows 2003 servers. It's fast, it's slick, it allows copy-paste and with a bit of command line fu you can connect to the physical display (instead of one of the two virtual sessions). It also uses built-in Windows authentication and offers encryption.

So, now that I'm rolling out CentOS 5 servers - I need something similar that allows me to look at the graphical UI on the box from elsewhere. From what I can tell, my options are:

KVM that supports TCP/IP

Probably one of the holy grails of remote administration. It allows you to see everything from the BIOS setup screen onward without needing to be physically at the machine. The downside is cost. So while I will eventually be hooking one of these up, it's not in the budget for this quarter.

VNC over SSH

I have a love/hate relationship with VNC. On the Windows clients, we use UltraVNC with built-in Windows authentication and the AES encryption plug-in.

But if you want to wrap VNC with SSH, you have to configure port forwarding all the time in PuTTY. Which turns connecting to a remote server into a multi-step process. With Windows' RDP, I just say "connect to IP address X" and I'm done (and I can connect in as anyone that I want). For PuTTY+VNC, I have to jump through a lot more hoops.

There's also the (possible) issue that VNC is nowhere as efficient over the network as RDP. Once you use Terminal Services' RDP, you'll be spoiled and never want to use older technologies. It (almost) never glitches, it's lightning fast and responsive, and it's just pure remote GUI goodness (except for being a MS-only protocol).

X11 over SSH

This is where I'm heading at the moment. It uses SSH for authentication, so we can lock things down that way (forcing the use of public keys).

Now, a word of caution. A misconfigured SSH or X11 server is a security breach waiting to happen. Pay close attention to chapter 9 in SSH, The Secure Shell, The Definitive Guide by Barrett, Silverman & Byrnes (published by O'Reilly).

Installing Xming on Windows

In order to do X11 on Microsoft Windows, you need to install "X Server" software on the Windows box. While there are pay options out there, I'd suggest starting with Xming which is free (GPLv2). You'll want to download and install both Xming and Xming-fonts.

Configuration of sshd and X11

In order for the local X Server (Xming - running on your Windows system) to talk to the remote Linux server, you'll need to verify some settings on the Linux server. First up is configuration of the sshd daemon (typically /etc/ssh/sshd_config for OpenSSH). Look for the following 2 lines and make sure they are configured correctly:

X11Forwarding yes
#X11UseLocalhost yes

By default, OpenSSH ships with X11Forwarding set to "no" but the default for X11UseLocalhost is "yes". So you should only have to add the "X11Forwarding yes" line.

Create a PuTTY session

I'll make the assumption that you're going to use a PuTTY public-key pair. If you need to install a generated PuTTY key (maybe you want to use a separate PuTTY key for X11 forwarding), then here are the directions for OpenSSH.

(login as yourself or as root and then "su" to your username)
# cd ~/.ssh
# cat > machinename@svn.pub
(paste in PuTTY key)
# ssh-keygen -i -f machinename@svn.pub >> authorized_keys
(Ctrl-D to exit)

  1. Right-click on the Pageant icon in the system tray and choose "New Session".
  2. Enter the hostname (i.e. 192.168.1.1)
  3. Go to the Connection -> SSH -> X11 tab
  4. Turn ON "X11 forwarding"
  5. Display location should be: localhost:0
  6. Go back to the Session tab
  7. Enter a name in the Saved Sessions text box (i.e. "MyHost-X11") and click on "Save"
  8. Click the "Open" button to connect to the server

If all goes well, you should see a line like:

/usr/bin/xauth: creating new authority file /home/thomas/.Xauthority

Which tells us that SSH is ready to do some X forwarding.

Fire up Xming

If you haven't already ran Xming you should run XLaunch and just roll through the defaults. Now, in the PuTTY window that is sitting at a command prompt, try:

# xeyes

And you should see the xeyes application open up on your Windows system. If you want to continue to start up other X applications, put an ampersand (&) at the end of the line.

More advanced stuff

  1. Fire up XLaunch
  2. Select "One window" and click "Next"
  3. Select "Start a program" and click "Next"
  4. The start program should be either "gnome-session" or "startkde"
  5. Select Run Remote using PuTTY (plink.exe) and turn on the compression option.
  6. Enter the IP address or hostname in "Connect to computer" of the Linux box that you are connecting to
  7. Enter your username in the "Login as user"
  8. Click the "Next" button
  9. In the "Additional parameters", enter "-screen 0 1024 768" which will set screen zero to be 1024x768
  10. If you run your SSH server on a non-standard port, enter "-P port" in the PuTTY extra options field (run "plink" at a Windows command prompt to see the possible options)
  11. Save your configuration file and click "Finish"

If all goes well, you should see the Gnome desktop!

Final thoughts (for the moment)

Now, it's still not as slick as Terminal Services. But it seems to work just fine and gives me a GUI desktop. I still plan on doing most of my administration from the command line, but this provides a nice GUI for those who follow in my footsteps.

Labels: , ,

UltraVNC (Server) Install on Windows XP
Installing UltraVNC (see also "UltraVNC Installation")

  1. Download UltraVNC for MS Windows
  2. Run the setup program (currently this is: UltraVNC-100-RC203-Setup.exe)
  3. Accept the license agreement and read the Information screen
  4. Use the default install destination location
  5. Choose "Complete Install"
  6. Use the default Start Menu Folder
  7. Turn ON "Register Ultr@VNC Server as system service"
  8. Turn ON "Start or restart Ultr@VNC service"
  9. Turn OFF the (3) options that create desktop icons
  10. Turn ON "Associated .vnc files with Ultr@VNC Viewer"
  11. Click "Install" to start the installation.

WinVNC: Default Local System Properties (see "configuration for details)

  1. Turn OFF "Enable JavaViewer"
  2. Turn ON "Display Query Window", Set the timeout to 60 seconds, with "Accept" as the default action.
  3. Under "Multi viewer connections", CHANGE to "Keep existing connections"
  4. Under "Authentication", set a secure default password
  5. Under "Authentication", turn ON "Require MS Logon", turn ON "New MS Logon"
  6. Click on "Configure MS Logon Groups", Add, enter "Administrators" (note the plural) and click "OK". Grant that group full control and click "OK" to close the UltraVNC Security Editor window.
  7. Most other options can be left "as is"

AES Encryption plugin (a.k.a. DSM)

  1. Download the AESV2 Plugin (currently: AESV2Plugin100.zip)
  2. Extract the .DSM file to the program folder where you installed UltraVNC (usually: C:\Program Files\UltraVNC), see "DSM quick start" for more information.
  3. Re-open the "Default Local System Properties" window for the UltraVNC server (Start -> UltraVNC -> UltraVNC Server -> Show Default Settings). Alternately, start up the service helper systray app (Run Service Helper) and go to "Admin Properties")
  4. Under "DSM Plugin", turn ON the "Use:" checkbox and select "AESV2Plugin.dsm" from the list.
  5. Click "OK"

Labels: , ,