Re: shutdown is taking more than about 15 minutes (372 Views)
Reply
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 1 of 18 (372 Views)
Accepted Solution

shutdown is taking more than about 15 minutes

Hi.
Linux Node1 2.6.10-telco-1.46-mckinley-smp #1 SMP Fri May 30 18:29:43 UTC 2008 ia64 GNU/Linux

I have above linux platform. When I issue shutdown, sometimes I find system is taking about 15 minutes after it had broadcasted the message about shutdown. I issue shutdown -r now
Please use plain text.
Honored Contributor
Oviwan
Posts: 692
Registered: ‎11-29-2005
Message 2 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

is there any cluster software running or any other software that has a long time to finish?
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 3 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

we have dual node software systems. By the way. shutdown should be taking care of it I believe. When system is issued shutdown it sends SIGTERM to all for graceful exit, if processes are still stuck up , then brutal SIGKILL is sent and reboot is forced. Why do you think in any case , shutdown should hang for 15 minutes ?
Please use plain text.
Honored Contributor
Wouter Jagers
Posts: 495
Registered: ‎10-01-2002
Message 4 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

Some things just won't die ;-)

Processes can be in an 'uninterruptable sleep', for example when performing I/O operations. They will then only receive the KILL when coming out of that state. Maybe you have a process blocked in such a state for a while. (your fencing mechanism, maybe?)

Can you see where it's hanging, exactly ? The console should be telling you what the machine is currently trying to do. It could be useful to provide the last few lines of console output you see when it hangs.

You can also try to pinpoint this by stopping services manually. Since the cluster software would be our prime suspect, see what happens when you manually shut down these services (using the init scripts in /etc/init.d) while the machine is still up.

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.
Please use plain text.
Honored Contributor
Wouter Jagers
Posts: 495
Registered: ‎10-01-2002
Message 5 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

Sorry, I forgot to ask an important question.

Are you using HP ServiceGuard for linux or another clustering solution ?

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 6 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

Its linux system.
I did not observe any console message flashed after shutdown issued. No console messages for 15 minutes. Do you suggest any degug I should add? Like
1. Adding "top" output to see , if any process is hung

2. Adding ps output and check out the status of processes after shutdown command is issued.

If you have any such specific debug command that I can insert in my perl code to find out which processes goes in uninterruptable sleep, that would be great help

Another question , why would a process goes for uninterruptable sleep ? KILL -9 also can not kill them ? though I have seen many a times, kill -9 too does not help in killing out the processes

Thanks
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 7 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

Hey...
I have one important question. Will shutdown hang , if any of the process does not exit , or hung. I thought "shutdown -r now" would anyway proceed for rebooting, if its not able to kill any of the process that are hung.

Pls clerify , if my concept is right or wrong. Describe how shutdown works in steps.
Please use plain text.
Honored Contributor
skt_skt
Posts: 1,875
Registered: ‎12-27-2006
Message 8 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

"I did not observe any console message flashed after shutdown issued"

r u really wathcing the console (not the terminal you issued the shutdown). Or try watching the /var/log/boot.log or /var/log/messages from another terminal parallay.
Please use plain text.
Honored Contributor
Wouter Jagers
Posts: 495
Registered: ‎10-01-2002
Message 9 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

Why would a process go into such an uninterruptable state: it mustn't get killed in the middle of an I/O operation, because that could cause corruption. However, a process is supposed to be in such a state for only very small amounts of time, unless if it's blocked for some reason. It will not ignore the kill -9, it will just only see it when it returns from the operation.

Thay only illustrates why a process would not react to a kill -9 immediately, though.

The man page for shutdown is quite short and describes the shutdown process well.

Are you working on the system console, or through telnet/ssh ?

We still need to know which cluster software you are using.

What's the output of these commands:

# whereis cmviewcl
# service cman status

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 10 of 18 (372 Views)

Re: shutdown is taking more than about 15 minutes

service cman status
-su: service: command not found

service tcp status
-su: service: command not found

whereis cmviewcl
cmviewcl:

I have console connection to platform. Its not telnet / ssh sessions. thats for sure.

I did not find boot.log file under /var/log

kern.log
Jul 7 02:29:05 Node1 kernel: EXT2-fs warning: maximal mount count reached, running e2fsck is recommended
Jul 7 02:45:18 Node1 kernel: nfsd: last server has exited
Jul 7 02:45:18 Node1 kernel: nfsd: unexporting all filesystems

Shutdown was issued around 02:29:46

There is no messages in syslog and kern.log around 15 mins after shutdown was invoked.

Thanks
Please use plain text.
Honored Contributor
Wouter Jagers
Posts: 495
Registered: ‎10-01-2002
Message 11 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

ok, no recent redhat.

We're really going to need more info on your machine. The linux distribution you're running as well as the software are important to know in case of problems.

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.
Please use plain text.
Honored Contributor
Paul McCleary
Posts: 392
Registered: ‎11-03-1999
Message 12 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

What distribution are you using and how up to date is it?

Seems like you could be having a problem with NFS.

Can you replicate the same delay you have by just stopping NFS instead of running a shutdown?
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 13 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

Hey I don't think I am using any of NFS file systems
ii acpid 1.0.4-1 Utilities for using ACPI power management
ii addoc 13.00.39 This package is a wrapper to install OpenCall
ii adduser 3.63 Add and remove users and groups
ii apt 0.5.28.6 Advanced front-end for dpkg
ii apt-utils 0.5.28.6 APT utility programs
ii aptitude 0.2.15.9-2 terminal-based apt frontend
ii ash 0.5.2-5 Compatibility package for the Debian Almquist Shell
ii base-config 2.53.10.2-0.0.0.1.pure64 Debian base system configurator
ii base-files 3.1.2 Debian base system miscellaneous files
ii base-passwd 3.5.9 Debian base system master password and group files
ii bash 2.05b-26 The GNU Bourne Again SHell
ii bash3 3.0-12 The GNU Bourne Again SHell (Version 3)
ii bash3-doc 3.0-12 Documentation and examples for the The GNU Bourne Again SHell
ii bc 1.06-15 The GNU bc arbitrary precision calculator language
ii bind9-host 9.2.4-1sarge3 Version of 'host' bundled with BIND 9.X
ii binutils 2.15-6 The GNU assembler, linker and binary utilities
ii bison 1.875d-1 A parser generator that is compatible with YACC
ii bmclog 3.4 log BMC events to the evlog daemon
ii branding-hp-telco 2.22.4 branding files for HP Telco product
ii bsdmainutils 6.0.17 collection of more utilities from FreeBSD
ii bsdutils 2.12p-4sarge1 Basic utilities from 4.4BSD-Lite
ii build-essential 10.1 informational list of build-essential packages
ii bzip2 1.0.2-7 high-quality block-sorting file compressor - utilities
ii console-common 0.7.49 Basic infrastructure for text console configuration
ii console-data 2002.12.04dbs-49 Keymaps, fonts, charset maps, fallback tables for console-tools
ii console-tools 0.2.3dbs-56 Linux console and font utilities
ii coreutils 5.2.1-2 The GNU core utilities
ii cpio 2.5-1.3 GNU cpio -- a program to manage archives of files.
ii cpp 3.3.5-3 The GNU C preprocessor (cpp)
ii cpp-3.3 3.3.5-13 The GNU C preprocessor
ii cpp-3.4 3.4.3-13sarge1 The GNU C preprocessor
ii cramfsprogs 1.1-6 Tools for CramFs (Compressed ROM File System)
ii crash 4.0-3.7-0hpde1 kernel debugging utility, allowing gdb like syntax
ii cron 3.0pl1-86 management of regular background processing
ii dash 0.5.2-5 The Debian Almquist Shell
ii dc 1.06-15 The GNU dc arbitrary precision reverse-polish calculator
ii debconf 1.4.30.13 Debian configuration management system
ii debconf-i18n 1.4.30.13 full internationalization support for debconf
ii debianutils 2.8.4 Miscellaneous utilities specific to Debian
ii defoma 0.11.8-0.1 Debian Font Manager -- automatic font configuration framework
ii dhcp 2.0pl5-19.1sarge2 DHCP server for automatic IP address assignment
ii dhcp-client 2.0pl5-19.1sarge2 DHCP Client
ii dhcp-config 22.00.032 DHCP configuration
ii diff 2.8.1-11 File comparison utilities
ii dnsutils 9.2.4-1sarge3 Clients provided with BIND
ii doc-debian 3.1 Debian Project documentation, Debian FAQ and other documents
ii doc-linux-text 2005.04-1 Linux HOWTOs and FAQs in ASCII format
ii dosfstools 2.11-2.1 Utilities to create and check MS-DOS FAT filesystems
ii dpkg 1.10.28 Package maintenance system for Debian
ii dpkg-dev 1.10.28 Package building tools for Debian
ii dselect 1.10.28 a user tool to manage Debian packages
ii dumputils 0.9.6-0hpde1 simple configuration and dump recovery utilities for LKCD
ii e2fslibs 1.37-2sarge1 ext2 filesystem libraries
ii e2fsprogs 1.37-2sarge1 ext2 file system utilities and libraries
ii ed 0.2-20 The classic unix line editor
ii efibootmgr 0.4.9-0.sarge.2 Interact with the EFI Boot Manager
ii eject 2.0.13deb-8sarge2 ejects CDs and operates CD-Changers under Linux
ii elilo 3.4-7.sarge.1 Bootloader for systems using EFI-based firmware
ii emacsen-common 1.4.16 Common facilities for all emacsen
ii ethtool 2-1 Display or change ethernet card settings
ii evlog 1.6.1-0hpde3 Enterprise Event Logging tools and daemon
ii evlog-dev 1.6.1-0hpde3 Headers and libraries for developing apps that use Evlog POSIX
ii expect 5.42.1-1.2 A program that "talks" to other programs
ii expect-dev 5.42.1-1.2 Development files for the expect package
ii fbset 2.1-16 framebuffer device maintenance program
ii fdutils 5.4-20040228-1 Linux floppy utilities
ii file 4.12-1sarge2 Determines file type using "magic" numbers
ii fileutils 5.2.1-2 The GNU file management utilities (transitional package)
ii findutils 4.1.20-6 utilities for finding files--find, xargs, and locate
ii flex 2.5.31-31sarge1 A fast lexical analyzer generator.
ii fontconfig 2.3.1-2 generic font configuration library
ii ftp 0.17-12 The FTP client.
ii ftpd 0.17-20sarge2 FTP server
ii g++ 3.3.5-3 The GNU C++ compiler
ii g++-3.3 3.3.5-13 The GNU C++ compiler
ii g++-3.4 3.4.3-13sarge1 The GNU C++ compiler
ii gawk 3.1.4-2 GNU awk, a pattern scanning and processing language
ii gcc 3.3.5-3 The GNU C compiler
ii gcc-3.3 3.3.5-13 The GNU C compiler
ii gcc-3.3-base 3.3.5-13 The GNU Compiler Collection (base package)
ii gcc-3.4 3.4.3-13sarge1 The GNU C compiler
ii gcc-3.4-base 3.4.3-13sarge1 The GNU Compiler Collection (base package)
ii gdb 6.4.90.dfsg-0hpde1 The GNU Debugger
ii gettext-base 0.14.4-2 GNU Internationalization utilities for the base system
ii glibc-doc 2.3.2.ds1-22sarge6hpde1 GNU C Library: Documentation
ii gnu-efi 3.0a-4 Library for developing EFI applications
ii gnupg 1.4.1-1.sarge7 GNU privacy guard - a free PGP replacement
ii gnustep-base-common 1.10.3-1 GNUstep Base library - common files
ii gnustep-make 1.10.0-5 Basic GNUstep Scripts and Makefiles
ii gnutls-bin 1.0.16-13.2sarge2 GNU TLS library - commandline utilities
ii grep 2.5.1.ds1-4 GNU grep, egrep and fgrep
ii groff-base 1.18.1.1-7 GNU troff text-formatting system (base system components)
ii gzip 1.3.5-10sarge2 The GNU compression utility
ii high-res-timer 1.3-1 User space header to access ITC on ia64
ii hostname 2.13 A utility to set/show the host name or domain name
ii hp-telco-ipmi 1.7 HP Telco IPMI configuration
ii hpde-support 1.4 Collect information useful for HP support engineers
ii hpmgmtbase 2.0.8-0hpde4 Hewlett Packard Management Base Utilities
ii hpoc-rsc-documentation 3.2.rsc.4.1-1 HP OpenCall RSC 3.0 Documentation
ii hpocconfiguration 3.2.rsc.4.1-1 HP OpenCall Configuration software
ii hpocha 3.2.rsc.4.1-1 HP OpenCall High-availability software
ii hpoclinux-wrapper 3.2.rsc.4.1-1 HP OpenCall OS Run-Time Environment
ii hpocnettl 3.2.rsc.4.1-1 HP OpenCall Network Tracing and Logging software
ii hpocoe-base-runtime 3.0-1.hpte Runtime bundle (task) package for HP OpenCall products
ii hpocrsc 3.2.rsc.4.1-3 HP OpenCall software
ii hpocsecurity 3.2.rsc.4.1-1 HP OpenCall Security software
ii hpocss7 3.2.rsc.4.1-1 HP OpenCall SS7 software
ii hpocss7-hw-enabled 3.2.rsc.4.1-1 HP OpenCall SS7 HW enabled software
ii hpocss7-hw-mgtlib 3.2.rsc.4.1-1 HP OpenCall SS7 Hardware Management Libs Package
ii hpocss7-hw-rmon 3.2.rsc.4.1-1 HP OpenCall SS7 HW enabled Stack Monitoring software
ii hpocss7drivers-source 3.2.rsc.4.1-1 HP OpenCall RSC Driver Source Package
ii hpocss7drivers-user-spac 3.2.rsc.4.1-2 HP OpenCall SS7 Hardware Package
ii hpocsupportability 3.2.rsc.4.1-1 HP OpenCall Supportability software
ii ifenslave-2.6 1.1.0-4 Attach and detach slave interfaces to a bonding device
ii ifrename 27-2 Rename network interfaces based on various static criteria
ii ifupdown 0.6.7 high level tools to configure network interfaces
ii info 4.7-2.2sarge2 Standalone GNU Info documentation browser
ii initrd-tools 0.1.81.1 tools to create initrd image for prepackaged Linux kernel
ii initscripts 2.86.ds1-1 Standard scripts needed for booting and shutting down
ii ipchains 1.3.10-15 Network firewalling for Linux 2.2.x
ii ipmasqadm 0.4.2-2 Utility for configuring extra masquerading functionality
ii iproute 20041019-3 Professional tools to control the networking in Linux kernels
ii iptables 1.2.11-10 Linux kernel 2.4+ iptables administration tools
ii iputils-arping 20020927-2 Tool to send ICMP echo requests to an ARP address
ii iputils-ping 20020927-2 Tools to test the reachability of network hosts
ii iputils-tracepath 20020927-2 Tools to trace the network path to a remote host
ii kernel-2.6.10-telco-1.43 3.2.rsc.4.1-3 HP OpenCall SS7 Hardware Package
ii kernel-headers-2.6.10-te 2.6.10-telco-1.43 HP Telco Extensions kernel headers
ii kernel-headers-2.6.10-te 2.6.10-telco-1.43 HP Telco Extensions kernel headers
ii kernel-headers-telco-mck 2.6.10-telco-1.43 HP Telco Extensions kernel headers
ii kernel-image-2.6.10-telc 2.6.10-telco-1.43 Linux kernel image for HP Itanium II Telco systems
ii kernel-image-2.6.10-telc 2.6.10-telco-1.43 Debuggable Linux kernel image for HP Itanium II Telco systems
ii kernel-image-telco-mckin 2.6.10-telco-1.43 HP Telco Extensions kernel image
ii kernel-kbuild-2.6-3 2.6.8-2 Linux kernel 2.6 kbuild tools
ii kexec-tools 1.101-1hpde3 kexec tool
ii klogd 1.4.1-17hpde2 Kernel Logging Daemon
ii ksymoops 2.4.9-1 Linux kernel oops and error message decoder
ii less 382-1 Pager program similar to more
ii libacl1 2.2.23-1 Access control list shared library
ii libappconfig-perl 1.56-2 Perl module for configuration file and command line handling
ii libarray-intspan-perl 2.001-1ocek1 a Module for handling arrays using IntSpan techniques
ii libatm1 2.4.1-17 shared library for ATM (Asynchronous Transfer Mode)
ii libattr1 2.4.16-1 Extended attribute shared library
ii libblkid1 1.37-2sarge1 block device id library
ii libbz2-1.0 1.0.2-7 high-quality block-sorting file compressor library - runtime
ii libbz2-dev 1.0.2-7 high-quality block-sorting file compressor library - development
ii libc6.1 2.3.2.ds1-22sarge6hpde1 GNU C Library: Shared libraries and Timezone data
ii libc6.1-dev 2.3.2.ds1-22sarge6hpde1 GNU C Library: Development Libraries and Header Files
ii libcap1 1.10-14 support for getting/setting POSIX.1e capabilities
ii libclass-data-inheritabl 0.02-1 Inheritable, overridable class data
ii libclass-methodmakerocbu 1.0809-1ocek1 A module for creating generic methods
ii libcomerr2 1.37-2sarge1 common error description library
ii libconsole 0.2.3dbs-56 Shared libraries for Linux console and font manipulation
ii libdb1-compat 2.1.3-7 The Berkeley database routines [glibc 2.0/2.1 compatibility]
ii libdb2 2.7.7.0-9 The Berkeley database routines (run-time files)
ii libdb3 3.2.9+dfsg-0.0hpde1 Berkeley v3 Database Libraries [runtime]
ii libdb4.1 4.1.25+0hp-1 Berkeley v4.1 Database Libraries [runtime]
ii libdb4.2 4.2.52+dfsg-0hpde1 Berkeley v4.2 Database Libraries [runtime]
ii libdevel-stacktrace-perl 1.11-1 Stack trace and stack trace frame objects
ii libdevmapper1.01 1.01.00-4sarge1 The Linux Kernel Device Mapper userspace library
ii libdns16 9.2.4-1sarge3 DNS Shared Library used by BIND
ii libdps1 4.3.0.dfsg.1-14sarge4 Display PostScript (DPS) client library
ii liberror-perl 0.15-6 Perl module for error/exception handling in an OO-ish way
ii libexception-class-perl 1.20-1 a module that allows you to declare real exception classes in Pe
ii libexpat1 1.95.8-3 XML parsing C library - runtime library
ii libffcall1 1.10-2 Foreign Function Call Libraries
ii libfontconfig1 2.3.1-2 generic font configuration library (shared library)
ii libfreetype6 2.1.7-8 FreeType 2 font engine, shared library files
ii libgcc1 3.4.3-13sarge1 GCC support library
ii libgcrypt11 1.2.0-11.1 LGPL Crypto library - runtime library
ii libgdbm3 1.8.3-2 GNU dbm database routines (runtime version)
ii libglib1.2 1.2.10-9 The GLib library of C routines
ii libgnustep-base1.10 1.10.3-1 GNUstep Base library
ii libgnutls11 1.0.16-13.2sarge2 GNU TLS library - runtime library
ii libgpg-error0 1.0-1 library for common error values and messages in GnuPG components
ii libgtk1.2 1.2.10-17 The GIMP Toolkit set of widgets for X
ii libgtk1.2-common 1.2.10-17 Common files for the GTK+ library
ii libice6 4.3.0.dfsg.1-14sarge4 Inter-Client Exchange library
ii libidn11 0.5.13-1.0 GNU libidn library, implementation of IETF IDN specifications
ii libio-stringy-perl 2.110-1 Perl5 modules for IO from scalars and arrays
ii libisc7 9.2.4-1sarge3 ISC Shared Library used by BIND
ii libiw27 27-2 Wireless tools - library
ii libjpeg62 6b-10 The Independent JPEG Group's JPEG runtime library
ii libjudydebian1 1.0.2-0hpde1 C library for creating and accessing dynamic arrays
ii libldap2 2.1.30-8 OpenLDAP libraries
ii liblocale-gettext-perl 1.01-17 Using libc functions for internationalization in Perl
ii liblockfile1 1.06 NFS-safe locking library, includes dotlockfile program
ii liblwres1 9.2.4-1sarge3 Lightweight Resolver Library used by BIND
ii liblzo1 1.08-1.2 A real-time data compression library
ii libmagic1 4.12-1sarge2 File type determination library using "magic" numbers
ii libncurses5 5.4-4 Shared libraries for terminal handling
ii libnet-netmask-perl 1.9011-1 parse, manipulate and lookup IP network blocks
ii libnet-telnet-perl 3.03-1 Script telnetable connections
ii libnewt0.51 0.51.6-20 Not Erik's Windowing Toolkit - text mode windowing with slang
ii libobjc1 3.3.5-13 Runtime library for GNU Objective-C applications
ii libopencdk8 0.5.5-10 Open Crypto Development Kit (OpenCDK) (runtime)
ii libopenipmi0 1.3.16-1hpde2 Intelligent Platform Management Interface - runtime
ii libpam-modules 0.76-22 Pluggable Authentication Modules for PAM
ii libpam-runtime 0.76-22 Runtime support for the PAM library
ii libpam0g 0.76-22 Pluggable Authentication Modules library
ii libparse-recdescent-perl 1.94-4 Generates recursive-descent parsers in Perl
ii libparted1.6-12 1.6.21-1 The GNU Parted disk partitioning shared library
ii libpcap0.7 0.7.2-7 System interface for user-level packet capture
ii libpcap0.8 0.8.3-5 System interface for user-level packet capture
ii libpcre3 4.5-1.2sarge1 Perl 5 Compatible Regular Expression Library - runtime files
ii libphysmemmap 1.2 physical memory map library (accesses memory map in /proc/iomem)
ii libpng12-0 1.2.8rel-1 PNG library - runtime
ii libpopt0 1.7-5 lib for parsing cmdline parameters
ii libreadline4 4.3-11 GNU readline and history libraries, run-time libraries
ii libreadline5 5.0-10 GNU readline and history libraries, run-time libraries
ii libsasl2 2.1.19.dfsg1-0sarge2 Authentication abstraction library
ii libsctp-dev 1.0.2-1hpde3 user-space access to Linux Kernel SCTP - development files
ii libsctp1 1.0.2-1hpde3 user-space access to Linux Kernel SCTP - shared library
ii libsemaphore 0.1-4 posix-1b semaphore library
ii libsemaphore-dev 0.1-4 posix-1b semaphore development library
ii libsigc++-1.2-5c102 1.2.5-4 type-safe Signal Framework for C++ - runtime
ii libsm6 4.3.0.dfsg.1-14sarge4 X Window System Session Management library
ii libsnmp-base 5.1.2-6.2hpde1 NET SNMP (Simple Network Management Protocol) MIBs and Docs
ii libsnmp5 5.1.2-6.2hpde1 NET SNMP (Simple Network Management Protocol) Library
ii libss2 1.37-2sarge1 command-line interface parsing library
ii libssl0.9.6 0.9.6m-1sarge4 SSL shared libraries (old version)
ii libssl0.9.7 0.9.7e-3sarge4 SSL shared libraries
ii libstdc++5 3.3.5-13 The GNU Standard C++ Library v3
ii libstdc++5-3.3-dev 3.3.5-13 The GNU Standard C++ Library v3 (development files)
ii libstdc++6 3.4.3-13sarge1 The GNU Standard C++ Library v3
ii libstdc++6-dev 3.4.3-13sarge1 The GNU Standard C++ Library v3 (development files)
ii libstruct-compare-perl 1.0.1-3 Recursive diff for perl structures.
ii libtasn1-0 0.1.2-5 Manage ASN.1 structures (runtime)
ii libtasn1-2 0.2.10-3sarge1 Manage ASN.1 structures (runtime)
ii libtext-charwidth-perl 0.04-1 get display widths of characters on the terminal
ii libtext-iconv-perl 1.2-3 Convert between character sets in Perl
ii libtext-template-perl 1.44-1.1 Text::Template perl module
ii libtext-wrapi18n-perl 0.06-1 internationalized substitute of Text::Wrap
ii libunwind7 0.98.5-1 A library to determine the call-chain of a program - runtime
ii libunwind7-dev 0.98.5-1 A library to determine the call-chain of a program - development
ii libusb-0.1-4 0.1.10a-9.sarge.1 userspace USB programming library
ii libuuid1 1.37-2sarge1 universally unique id library
ii libwrap0 7.6.dbs-8 Wietse Venema's TCP wrappers library
ii libx11-6 4.3.0.dfsg.1-14sarge4 X Window System protocol client library
ii libxaw7 4.3.0.dfsg.1-14sarge4 X Athena widget set library
ii libxcursor1 1.1.3-1 X cursor management library
ii libxext6 4.3.0.dfsg.1-14sarge4 X Window System miscellaneous extension library
ii libxft1 4.3.0.dfsg.1-14sarge4 FreeType-based font drawing library for X (version 1)
ii libxft2 2.1.7-1 FreeType-based font drawing library for X
ii libxi6 4.3.0.dfsg.1-14sarge4 X Window System Input extension library
ii libxml2 2.6.16-7 GNOME XML library
ii libxml2-dev 2.6.16-7 Development files for the GNOME XML library
ii libxmu6 4.3.0.dfsg.1-14sarge4 X Window System miscellaneous utility library
ii libxmuu1 4.3.0.dfsg.1-14sarge4 lightweight X Window System miscellaneous utility library
ii libxp6 4.3.0.dfsg.1-14sarge4 X Window System printing extension library
ii libxpm4 4.3.0.dfsg.1-14sarge4 X pixmap library
ii libxrandr2 4.3.0.dfsg.1-14sarge4 X Window System Resize, Rotate and Reflection extension library
ii libxrender1 0.8.3-7 X Rendering Extension client library
ii libxslt1.1 1.1.12-8 XSLT processing library - runtime library
ii libxt6 4.3.0.dfsg.1-14sarge4 X Toolkit Intrinsics
ii libxtrap6 4.3.0.dfsg.1-14sarge4 X Window System protocol-trapping extension library
ii libxtst6 4.3.0.dfsg.1-14sarge4 X Window System event recording and testing extension library
ii libxv1 4.3.0.dfsg.1-14sarge4 X Window System video extension library
ii linux-kernel-headers 2.5.999-test7-bk-17hpde1 Linux Kernel Headers for development
ii lksctp-tools 1.0.2-1hpde3 user-space access to Linux Kernel SCTP - commandline tools
ii locales 2.3.2.ds1-22sarge6hpde1 GNU C Library: National Language (locale) data [support]
ii login 4.0.3-31sarge9 system login tools
ii logrotate 3.7-5 Log rotation utility
ii lsb-release 1.4-8 LSB release command
ii lsof 4.74.dfsg.3-2 List open files.
ii lvm-common 1.5.17 The Logical Volume Manager for Linux (common files)
ii lvm2 2.01.04-5hpde1 The Linux Logical Volume Manager
ii m4 1.4.2-1 a macro processing language
ii make 3.80-9 The GNU version of the "make" utility.
ii makedev 2.3.1-77 creates device files in /dev
ii man-db 2.4.2-21sarge1 The on-line manual pager
ii manpages 1.70-1 Manual pages about using a GNU/Linux system
ii manpages-dev 1.70-1 Manual pages about using GNU/Linux for development
ii mawk 1.3.3-11 a pattern scanning and text processing language
ii mdadm 1.9.0-4sarge1 Manage MD devices aka Linux Software Raid
ii mii-diag 2.09-1 A little tool to manipulate network cards
ii minicom 2.1-9 friendly menu driven serial communication program
ii mkisofs 2.01+01a01-2 Creates ISO-9660 CD-ROM filesystem images
ii mm2-aaa-os-configuration 22.00.032 OS Configuration package
ii mm2-fgtod 22.00.032 SCP Fast Get Time Of Day library
ii mm2-firmware 22.00.032 firmware upgrade for rx2600 and cx2600
ii mm2-ha 22.00.032 SCP team high availability software
ii mm2-pt 22.00.032 Common controller platform applications and utilities
ii modconf 0.2.48 Device Driver Configuration
ii module-init-tools 3.2-pre1-2 tools for managing Linux kernel modules
ii modutils 2.4.26-1.2 Linux module utilities
ii mount 2.12p-4sarge1 Tools for mounting and manipulating filesystems
ii mpack 1.6-1 tools for encoding/decoding MIME messages
ii mtools 3.9.9-2.1 Tools for manipulating MSDOS files
ii mtr-tiny 0.67-1 Full screen ncurses traceroute tool
ii nano 1.2.4-5 free Pico clone with some new features
ii ncurses-base 5.4-4 Descriptions of common terminal types
ii ncurses-bin 5.4-4 Terminal-related programs and man pages
ii ncurses-term 5.4-4 Additional terminal type definitions
ii net-tools 1.60hpde-10-2 The NET-3 networking toolkit
ii netbase 4.21 Basic TCP/IP networking system
ii netkit-inetd 0.10-10 The Internet Superserver
ii nfs-common 1.0.6-3.1 NFS support files common to client and server
ii nfs-kernel-server 1.0.6-3.1 Kernel NFS server support
ii nscd 2.3.2.ds1-22sarge6hpde1 GNU C Library: Name Service Cache Daemon
ii ntp 4.2.0a+stable-2sarge1 Network Time Protocol: network utilities
ii ntp-refclock 4.2.0a+stable-2sarge1 Network Time Protocol: daemon for reference clocks
ii ntp-server 4.2.0a+stable-2sarge1 Network Time Protocol: common server tools
ii ntpdate 4.2.0a+stable-2sarge1 The ntpdate client for setting system time from NTP servers
ii nvi 1.79-22 4.4BSD re-implementation of vi
ii omni K269.02SP5EECNK.OMNI.ECN Signalware System
ii omni-a3 K269.02SP5EECNK.OMNIA3.E Signalware ANSI SS7 MTP Drivers
ii omni-a4 K269.02SP5EECNK.OMNIA4.E Signalware ANSI SS7 SCCP and TCAP Drivers
ii omni-a4s K269.02SP5EECNK.OMNIA4S. Signalware ANSI SCCP User Adaptation Layer Drivers
ii omni-c3 K269.02SP5EECNK.OMNIC3.E Signalware ITU SS7 MTP Drivers
ii omni-c4 K269.02SP5EECNK.OMNIC4.E Signalware ITU SS7 SCCP and TCAP Drivers
ii omni-c4s K269.02SP5EECNK.OMNIC4S. Signalware ITU SCCP User Adaptation Layer Drivers
ii omni-ch3 K269.02SP5EECNK.OMNICH3. Signalware Chinese SS7 MTP Drivers
ii omni-ch4 K269.02SP5EECNK.OMNICH4. Signalware Chinese SS7 SCCP and TCAP Drivers
ii omni-ch4s K269.02SP5EECNK.OMNICH4S Signalware Chinese SCCP User Adaptation Layer Drivers
ii omni-j3 K269.02SP5EECNK.OMNIJ3.E Signalware Japanese SS7 MTP Drivers
ii omni-j4 K269.02SP5EECNK.OMNIJ4.E Signalware Japanese SS7 SCCP and TCAP Drivers
ii omni-j4s K269.02SP5EECNK.OMNIJ4S. Signalware Japanese SCCP User Adaptation Layer Drivers
ii omni-man K269.02SP5EECNK.OMNIMAN. Signalware Manpages
ii omni-sctp K269.02SP5EECNK.OMNISCTP Signalware Stream Control Transmission Protocol Layer
ii omni-util K269.02SP5EECNK.OMNIUTIL Signalware Staged Installation Utility
ii openipmi 1.3.16-1hpde2 Intelligent Platform Management Interface (for servers)
ii parted 1.6.21-1 The GNU Parted disk partition resizing program
ii passwd 4.0.3-31sarge9 change and administer password and group data
ii patch 2.5.9-2 Apply a diff file to an original
ii pciutils 2.1.11-15 Linux PCI Utilities
ii pdksh 5.2.14-18 A public domain version of the Korn shell
ii perl 5.8.4-8sarge5 Larry Wall's Practical Extraction and Report Language
ii perl-base 5.8.4-8sarge5 The Pathologically Eclectic Rubbish Lister
ii perl-modules 5.8.4-8sarge5 Core Perl modules
ii pidentd 3.0.18-2 TCP/IP IDENT protocol server with DES support
ii portmap 5-9 The RPC portmapper
ii prctl 1.4-1 Process control operations
ii priocntl 1.4 Tool to manipulate process priorities and scheduling classes
ii procps 3.2.1-2hpde1 The /proc file system utilities
ii psmisc 21.5-1 Utilities that use the proc filesystem
ii quagga 0.98.3-7.4 unoff. successor of the Zebra BGP/OSPF/RIP routing daemon
ii quagga-doc 0.98.3-7.4 documentation files for quagga
ii rcs 5.7-15 The GNU Revision Control System
ii rsh-client 0.17-13 rsh clients.
ii rsh-server 0.17-13 rsh servers.
ii rsync 2.6.4-6 fast remote file copy program (like rcp)
ii salinfo 0.7-2 Monitor and decode events from Itanium firmware
ii sctp-cksumcfg 1.0.4 Configures SCTP checksum when SCTP is compiled in the kernel
ii sctp-sysctl 1.0.5.1 Installs SCTP specific sysctl tunables for telco environments
ii sctpspray 2.0.0-2 A utility to send one-way stream of SCTP (Stream Control
ii sctpstat 2.0.0-3 a utility to display detailed sctp information
ii sed 4.1.2-8 The GNU sed stream editor
ii sharutils 4.2.1-13 shar, unshar, uuencode, uudecode
ii shellutils 5.2.1-2 The GNU shell programming utilities (transitional package)
ii slang1 1.4.9dbs-8 The S-Lang programming library - runtime version
ii slang1a-utf8 1.4.9dbs-8 The S-Lang programming library with utf8 support
ii snmp 5.1.2-6.2hpde1 NET SNMP (Simple Network Management Protocol) Apps
ii snmpd 5.1.2-6.2hpde1 NET SNMP (Simple Network Management Protocol) Agents
ii ssh 3.8.1p1-8.sarge.6 Secure rlogin/rsh/rcp replacement (OpenSSH)
ii strace 4.5.8-1.2 A system call tracer
ii sudo 1.6.8p7-1.4 Provide limited super user privileges to specific users
ii sysklogd 1.4.1-17hpde2 System Logging Daemon
ii sysprofile 0.3.8 Modularized system wide shell configuration mechanism
ii sysstat 5.0.6-5 sar, iostat and mpstat - system performance tools for Linux
ii systemconfigurator 2.0.10-1 Unified Configuration API for Linux Installation
ii systemimager-client 3.2.3-6sarge4 Utilities for creating an image and upgrading client machines
ii systemimager-common 3.2.3-6sarge4 Utilities and libraries common to both the server and client
ii sysv-rc 2.86.ds1-1 Standard boot mechanism using symlinks in /etc/rc?.d
ii sysvinit 2.86.ds1-1 System-V like init
ii tar 1.14-2.3 GNU tar
ii task-hp-telco 5.4.1 HP Telco Extensions
ii tasksel 2.24 Tool for selecting tasks for installation on Debian system
ii tcl8.3 8.3.5-4 Tcl (the Tool Command Language) v8.3 - run-time files
ii tcl8.3-dev 8.3.5-4 Tcl (the Tool Command Language) v8.3 - development files
ii tcl8.3-doc 8.3.5-4 Tcl (the Tool Command Language) v8.3 - manual pages
ii tcl8.4 8.4.9-1 Tcl (the Tool Command Language) v8.4 - run-time files
ii tcl8.4-dev 8.4.9-1 Tcl (the Tool Command Language) v8.4 - development files
ii tclx8.3 8.3.5-4.1 Extended Tcl (TclX) version 8.3.5 -- TclX runtime package
ii tclx8.3-dev 8.3.5-4.1 Extended Tcl (TclX) version 8.3.5 -- TclX development package
ii tclx8.3-doc 8.3.5-4.1 Extended Tcl (TclX) version 8.3.5 -- TclX manpages
ii tcpd 7.6.dbs-8 Wietse Venema's TCP wrapper utilities
ii tcpdump 3.8.3-5sarge3 A powerful tool for network monitoring and data acquisition
ii tcsh 6.13.00-1 TENEX C Shell, an enhanced version of Berkeley csh
ii telnet 0.17-29 The telnet client
ii telnetd 0.17-29 The telnet server
ii texinfo 4.7-2.2sarge2 Documentation system for on-line information and printed output
ii textutils 5.2.1-2 The GNU text file processing utilities (transitional package)
ii time 1.7-21 The GNU time program for measuring cpu resource usage
ii timezoneconf 0.10 debconf interface to system timezone, date, and time
ii traceroute 1.4a12-18 traces the route taken by packets over a TCP/IP network
ii ttf-bitstream-vera 1.10-3 The Bitstream Vera family of free TrueType fonts
ii ucf 1.17 Update Configuration File: preserves user changes to config file
ii util-linux 2.12p-4sarge1 Miscellaneous system utilities
ii util-linux-locales 2.12p-4sarge1 Locales files for util-linux
ii whiptail 0.51.6-20 Displays user-friendly dialog boxes from shell scripts
ii whois 4.7.5 the GNU whois client
ii xbase-clients 4.3.0.dfsg.1-14sarge4 miscellaneous X clients
ii xfree86-common 4.3.0.dfsg.1-14sarge4 X Window System (XFree86) infrastructure
ii xinetd 2.3.13-3hpde1 replacement for inetd with many enhancements
ii xlibmesa-gl 4.3.0.dfsg.1-14sarge4 Mesa 3D graphics library [XFree86]
ii xlibmesa-glu 4.3.0.dfsg.1-14sarge4 Mesa OpenGL utility library [XFree86]
ii xlibs 4.3.0.dfsg.1-14sarge4 X Keyboard Extension (XKB) configuration data and metapackage
ii xlibs-data 4.3.0.dfsg.1-14sarge4 X Window System client data
ii xterm 4.3.0.dfsg.1-14sarge4 X terminal emulator
ii zlib1g 1.2.2-4.sarge.2hpde1 compression library - runtime
ii zlib1g-dev 1.2.2-4.sarge.2hpde1 compression library - development


1. More than above list of packages. What I am interested how to debug this issue. Is there somekind of "while loop" I could execute in loop for ps command , and determine which processes are not killed after shutdown was executed

2. Or I can strace on shutdown pid and determine how many processes that shutdown is waiting on before actually bringing donw the system

Pls suggest How we can debug in shutdown/init front, so that I can determine which process was hung for 15 minutes.
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 14 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

lsmod
Module Size Used by
bond0 160698 0
sctp 448112 6 [unsafe]
ipv6 650568 17 sctp
capability 15208 0
commoncap 21336 1 capability
thermal 38836 0
fan 16664 0
button 21488 0
processor 44344 1 thermal
ipmi_devintf 22364 2
ipmi_si 76484 1
ipmi_msghandler 108740 2 ipmi_devintf,ipmi_si
scsi_dump 20568 0
dump_blockdev 32364 1 scsi_dump
dump_gzip 13944 0
zlib_deflate 56820 1 dump_gzip
dump 100740 2 dump_blockdev,dump_gzip
rpqiosmp 8994240 0
e1000 317856 0
tg3 196480 0
e100 92478 0
tulip 119596 0
cmd64x 30272 0 [permanent]
loop 41576 0
vfat 36910 0
msdos 26954 0
fat 90164 2 vfat,msdos
isofs 65840 0
sr_mod 48096 0
ide_cd 90407 0
ide_core 276020 2 cmd64x,ide_cd
cdrom 90712 2 sr_mod,ide_cd
dm_mod 143152 17
raid1 45856 1
md 104416 2 raid1

Above is lsmod output , it may help you
Please use plain text.
Honored Contributor
Wouter Jagers
Posts: 495
Registered: ‎10-01-2002
Message 15 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

Aha, so you're on a debian box. Good to know.

Can you try to stop services manually, one by one ?
You could also try to change from runlevel 5 to runlevel 3, and then down to 1.. maybe you bump in to the hang then.

Note that runlevel 1 is a maintenance mode, no remote access will be possible.

# init 3

[..wait..]

# init 1

Just a thought..

Cheers,
Wout
an engineer's aim in a discussion is not to persuade, but to clarify.
Please use plain text.
Honored Contributor
Rob Leadbeater
Posts: 3,582
Registered: ‎08-14-2002
Message 16 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

Hi,

What applications are you running on the box ?

I've seen Oracle databases take ages (hours) to shut down cleanly, especially if they're busy.

After you've issued the shutdown -r command, take a look at what's running with top or similar. You should be able to work out what's not dying.

Cheers,

Rob
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 17 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

Hi
Finally we traced down wall was hung on tty_write call ,explananation is as below.
Please let me know what is the root cause for tty_write to get hung for about 15 minutes. This wall was invoked from shutdown.

wall hangs on tty_write system call for about 15 minutes on debian 2.6 kernel

--------------------------------------------------------------------------------

Hi
My system was about 20% cpu utilization. Console was open and some other terminals were oepn.

0. system info Linux Node1 2.6.10-telco-1.35.1-mckinley-smp #1 SMP Sun Nov 18 20:58:46 UTC 2007 ia64 GNU/Linux
1. System had invoked shutdown -r now commmand
2. As part of it , it was putting wall message to all ttys
The wall message is "The system is going down for reboot NOW!"
3. We found that wall was hanging for 15mins and was not posting this message to tty.
4. Because of above reason, shutdown was hanging for about 15 minutes.


===Code===
system ("while ( true );do sleep 25;date >> $rnps;ps -emo pid,ppid,psr,pcpu,pmem,vsz,rss,lwp,state,time,wchan=WIDE-WIDE-WCHAN-COLUMN,args>> $rnps; echo >> $rnps;done &");
...
..
shutdown -r now

====Code Ends===

Below is seen for about 15-mins as captured from ps -e
30133 30131 - 0.0 0.0 3632 1744 - - 00:00:00 -/usr/bin/wall
- - 3 0.0 - - - 30133 S 00:00:00 tty_write


Thanks
Krupa
Please use plain text.
Advisor
KRUPASINDHU PRADHAN
Posts: 19
Registered: ‎07-29-2008
Message 18 of 18 (371 Views)

Re: shutdown is taking more than about 15 minutes

Hi Rob,
I have ssh sessions open to this linux box. Do you expect this ssh sessions are hung for 15-minutes. Shutdown trying to put wall messatges to ssh terminals and got hung because of that ?

Appreciate your quick reply

Thanks
Krupa
Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation