Using sync command upon startup

[yoav_rubin2]

I'm using linux 4.1.8 32bit (yocto env) on p4080ds SoC.

On each startup the init process (systemV init) performs mount to a 15GB ext4 fs (nand flash).

Later (still by the init process and before getty invocation) the sync command is being executed.

Sometimes it hangs (takes about 20 sec) and sometimes not (run smoothly).

1. What is the reason for using the sync command upon system's **startup**? (even before a login is made).

2.  Can I safely remove the sync call upon startup? (I would like to do that in order for the system to always boot smoothly), Is it dangerous for any reason?

3. Where can I find some literature about the "backstage" operations of the sync system call in unix systems (specifically linux)

thanks a lot.  

[yipingwang]

Hello yoav rubin,

Syncing the disks(flash) could write corrupt data to the disks, such as if the panic was due to a bug in Ext4, for example - so the kernel plays it safe and doesn't flush anything to disk because it no longer trusts that the data is valid anymore.

Additionally, if the panic occurred due to a hardware issue (e.g. bad RAM), syncing the disks could also write corrupted data from memory to the disk. If the filesystem then became corrupt as a result, the system might refuse to boot after being restarted, or may need to be fsck'ed.

When safety is paramount mounting a filesystem with the sync option is always a good idea. This forces all writes to disk immediately - again this is a performance hit, but a good idea if you expect power failures or random strangers yanking the CF card out.

You could modify /etc/fstab to disable sync mount option.

In addition, for NAND flash UBI filesystem is recommended rather than ext4.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[yoav_rubin2]

thank you for your reply.

I'm sorry but I got a little bit confused:

1.

Syncing the disks(flash) could write corrupt data to the disks, such as if the panic was due to a bug in Ext4, for example - so the kernel plays it safe and doesn't flush anything to disk because it no longer trusts that the data is valid anymore.

2. 

Additionally, if the panic occurred due to a hardware issue (e.g. bad RAM), syncing the disks could also write corrupted data from memory to the disk. If the filesystem then became corrupt as a result, the system might refuse to boot after being restarted, or may need to be fsck'ed.

those two things you mentioned are bad possible outcomes for using sync at startup and yet we do have the sync command in our defaults scripts (rootfs image).

so is there any good reason at all to leave the sync command there and not cancel its invocation upon startup? 

another question plz:

the sync command called from inside "populate-volatile" script.

maybe you can help me understand what this script does (whats its purpose) and why it invokes the sync command (what is the sync's role serving this script's purpose)?

this is the script:

#!/bin/sh ### BEGIN INIT INFO # Provides:             volatile # Required-Start:       $local_fs # Required-Stop:      $local_fs # Default-Start:        S # Default-Stop: # Short-Description:  Populate the volatile filesystem ### END INIT INFO  # Get ROOT_DIR DIRNAME=`dirname $0` ROOT_DIR=`echo $DIRNAME | sed -ne 's:/etc/.*::p'`  [ -e ${ROOT_DIR}/etc/default/rcS ] && . ${ROOT_DIR}/etc/default/rcS # When running populate-volatile.sh at rootfs time, disable cache. [ -n "$ROOT_DIR" ] && VOLATILE_ENABLE_CACHE=no # If rootfs is read-only, disable cache. [ "$ROOTFS_READ_ONLY" = "yes" ] && VOLATILE_ENABLE_CACHE=no  CFGDIR="${ROOT_DIR}/etc/default/volatiles" TMPROOT="${ROOT_DIR}/var/volatile/tmp" COREDEF="00_core"  [ "${VERBOSE}" != "no" ] && echo "Populating volatile Filesystems."  create_file() {  EXEC=""  [ -z "$2" ] && {   EXEC="   touch \"$1\";   "  } || {   EXEC="   cp \"$2\" \"$1\";   "  }  EXEC="  ${EXEC}  chown ${TUSER}:${TGROUP} $1 || echo \"Failed to set owner -${TUSER}- for -$1-.\" >/dev/tty0 2>&1;  chmod ${TMODE} $1 || echo \"Failed to set mode -${TMODE}- for -$1-.\" >/dev/tty0 2>&1 "   test "$VOLATILE_ENABLE_CACHE" = yes && echo "$EXEC" >> /etc/volatile.cache.build   [ -e "$1" ] && {   [ "${VERBOSE}" != "no" ] && echo "Target already exists. Skipping."  } || {   if [ -z "$ROOT_DIR" ]; then    eval $EXEC   else    # Creating some files at rootfs time may fail and should fail,    # but these failures should not be logged to make sure the do_rootfs    # process doesn't fail. This does no harm, as this script will    # run on target to set up the correct files and directories.    eval $EXEC > /dev/null 2>&1   fi  } }  mk_dir() {  EXEC="  mkdir -p \"$1\";  chown ${TUSER}:${TGROUP} $1 || echo \"Failed to set owner -${TUSER}- for -$1-.\" >/dev/tty0 2>&1;  chmod ${TMODE} $1 || echo \"Failed to set mode -${TMODE}- for -$1-.\" >/dev/tty0 2>&1 "   test "$VOLATILE_ENABLE_CACHE" = yes && echo "$EXEC" >> /etc/volatile.cache.build  [ -e "$1" ] && {   [ "${VERBOSE}" != "no" ] && echo "Target already exists. Skipping."  } || {   if [ -z "$ROOT_DIR" ]; then    eval $EXEC   else    # For the same reason with create_file(), failures should    # not be logged.    eval $EXEC > /dev/null 2>&1   fi  } }  link_file() {  EXEC="  if [ -L \"$2\" ]; then   [ \"\$(readlink \"$2\")\" != \"$1\" ] && { rm -f \"$2\"; ln -sf \"$1\" \"$2\"; };  elif [ -d \"$2\" ]; then   if awk '\$2 == \"$2\" {exit 1}' /proc/mounts; then    cp -a $2/* $1 2>/dev/null;    cp -a $2/.[!.]* $1 2>/dev/null;    rm -rf \"$2\";    ln -sf \"$1\" \"$2\";   fi  else   ln -sf \"$1\" \"$2\";  fi         "   test "$VOLATILE_ENABLE_CACHE" = yes && echo " $EXEC" >> /etc/volatile.cache.build   if [ -z "$ROOT_DIR" ]; then   eval $EXEC  else   # For the same reason with create_file(), failures should   # not be logged.   eval $EXEC > /dev/null 2>&1  fi }  check_requirements() {  cleanup() {   rm "${TMP_INTERMED}"   rm "${TMP_DEFINED}"   rm "${TMP_COMBINED}"  }   CFGFILE="$1"   TMP_INTERMED="${TMPROOT}/tmp.$$"  TMP_DEFINED="${TMPROOT}/tmpdefined.$$"  TMP_COMBINED="${TMPROOT}/tmpcombined.$$"   sed 's@\(^:\)*:.*@\1@' ${ROOT_DIR}/etc/passwd | sort | uniq > "${TMP_DEFINED}"  cat ${CFGFILE} | grep -v "^#" | cut -s -d " " -f 2 > "${TMP_INTERMED}"  cat "${TMP_DEFINED}" "${TMP_INTERMED}" | sort | uniq > "${TMP_COMBINED}"  NR_DEFINED_USERS="`cat "${TMP_DEFINED}" | wc -l`"  NR_COMBINED_USERS="`cat "${TMP_COMBINED}" | wc -l`"   [ "${NR_DEFINED_USERS}" -ne "${NR_COMBINED_USERS}" ] && {   echo "Undefined users:"   diff "${TMP_DEFINED}" "${TMP_COMBINED}" | grep "^>"   cleanup   return 1  }    sed 's@\(^:\)*:.*@\1@' ${ROOT_DIR}/etc/group | sort | uniq > "${TMP_DEFINED}"  cat ${CFGFILE} | grep -v "^#" | cut -s -d " " -f 3 > "${TMP_INTERMED}"  cat "${TMP_DEFINED}" "${TMP_INTERMED}" | sort | uniq > "${TMP_COMBINED}"   NR_DEFINED_GROUPS="`cat "${TMP_DEFINED}" | wc -l`"  NR_COMBINED_GROUPS="`cat "${TMP_COMBINED}" | wc -l`"   [ "${NR_DEFINED_GROUPS}" -ne "${NR_COMBINED_GROUPS}" ] && {   echo "Undefined groups:"   diff "${TMP_DEFINED}" "${TMP_COMBINED}" | grep "^>"   cleanup   return 1  }   # Add checks for required directories here   cleanup  return 0 }  apply_cfgfile() {  CFGFILE="$1"  SKIP_REQUIREMENTS="$2"   [ "${VERBOSE}" != "no" ] && echo "Applying ${CFGFILE}"   [ "${SKIP_REQUIREMENTS}" == "yes" ] || check_requirements "${CFGFILE}" || {   echo "Skipping ${CFGFILE}"   return 1  }   cat ${CFGFILE} | sed 's/#.*//' | \  while read TTYPE TUSER TGROUP TMODE TNAME TLTARGET; do   test -z "${TLTARGET}" && continue   TNAME=${ROOT_DIR}${TNAME}   [ "${VERBOSE}" != "no" ] && echo "Checking for -${TNAME}-."    [ "${TTYPE}" = "l" ] && {    TSOURCE="$TLTARGET"    [ "${VERBOSE}" != "no" ] && echo "Creating link -${TNAME}- pointing to -${TSOURCE}-."    link_file "${TSOURCE}" "${TNAME}"    continue   }    [ "${TTYPE}" = "b" ] && {    TSOURCE="$TLTARGET"    [ "${VERBOSE}" != "no" ] && echo "Creating mount-bind -${TNAME}- from -${TSOURCE}-."    mount --bind "${TSOURCE}" "${TNAME}"    EXEC="  mount --bind \"${TSOURCE}\" \"${TNAME}\""    test "$VOLATILE_ENABLE_CACHE" = yes && echo "$EXEC" >> /etc/volatile.cache.build    continue   }    [ -L "${TNAME}" ] && {    [ "${VERBOSE}" != "no" ] && echo "Found link."    NEWNAME=`ls -l "${TNAME}" | sed -e 's/^.*-> \(.*\)$/\1/'`    echo ${NEWNAME} | grep -v "^/" >/dev/null && {     TNAME="`echo ${TNAME} | sed -e 's@\(.*\)/.*@\1@'`/${NEWNAME}"     [ "${VERBOSE}" != "no" ] && echo "Converted relative linktarget to absolute path -${TNAME}-."    } || {     TNAME="${NEWNAME}"     [ "${VERBOSE}" != "no" ] && echo "Using absolute link target -${TNAME}-."    }   }    case "${TTYPE}" in    "f")  [ "${VERBOSE}" != "no" ] && echo "Creating file -${TNAME}-."     TSOURCE="$TLTARGET"     [ "${TSOURCE}" = "none" ] && TSOURCE=""     create_file "${TNAME}" "${TSOURCE}" &     ;;    "d")  [ "${VERBOSE}" != "no" ] && echo "Creating directory -${TNAME}-."     mk_dir "${TNAME}"     # Add check to see if there's an entry in fstab to mount.     ;;    *)    [ "${VERBOSE}" != "no" ] && echo "Invalid type -${TTYPE}-."     continue     ;;   esac  done  return 0 }  clearcache=0 exec 9</proc/cmdline while read line <&9 do  case "$line" in   *clearcache*)  clearcache=1           ;;   *)        continue           ;;  esac done exec 9>&-  if test -e ${ROOT_DIR}/etc/volatile.cache -a "$VOLATILE_ENABLE_CACHE" = "yes" -a "x$1" != "xupdate" -a "x$clearcache" = "x0" then  sh ${ROOT_DIR}/etc/volatile.cache else  rm -f ${ROOT_DIR}/etc/volatile.cache ${ROOT_DIR}/etc/volatile.cache.build   # Apply the core file with out checking requirements. ${TMPROOT} is  # needed by check_requirements but is setup by this file, so it must be  # processed first and without being checked.  [ -e "${CFGDIR}/${COREDEF}" ] && apply_cfgfile "${CFGDIR}/${COREDEF}" "yes"   # Fast path: check_requirements is slow and most of the time doesn't  # find any problems. If there are a lot of config files, it is much  # faster to to concatenate them all together and process them once to  # avoid the overhead of calling check_requirements repeatedly  TMP_FILE="${TMPROOT}/tmp_volatile.$$"  rm -f "$TMP_FILE"   CFGFILES="`ls -1 "${CFGDIR}" | grep -v "^${COREDEF}\$" | sort`"  for file in ${CFGFILES}; do   cat "${CFGDIR}/${file}" >> "$TMP_FILE"  done   if check_requirements "$TMP_FILE"  then   apply_cfgfile "$TMP_FILE" "yes"  else   # Slow path: One or more config files failed requirements.   # Process each one individually so the offending one can be   # skipped   for file in ${CFGFILES}; do    apply_cfgfile "${CFGDIR}/${file}"   done  fi  rm "$TMP_FILE"   [ -e ${ROOT_DIR}/etc/volatile.cache.build ] && sync && mv ${ROOT_DIR}/etc/volatile.cache.build ${ROOT_DIR}/etc/volatile.cache fi  if [ -z "${ROOT_DIR}" ] && [ -f /etc/ld.so.cache ] && [ ! -f /var/run/ld.so.cache ] then  ln -s /etc/ld.so.cache /var/run/ld.so.cache fi

thanks again!

[yipingwang]

The populate-volatile.sh script is intended to handle all directories and files related to volatile storage.

[yoav_rubin2]

So this script creates directories and files related to volatile storage and invokes the sync command.

The sync command ensures that all writes to disks will be performed at this point.

why is it important that some of these files will be actually written to the disks at this point? they are regenerated upon each startup, so even if we will encounter sudden shutdown and lose these files they will be created again next time..

(I'm asking this because I'm trying to figure out the purpose of that sync call from this script upon startup)

If there is any literature/guide regarding this material I'll be happy to read it.

thanks!