Search:  
Gentoo Wiki

Emerge/Watch_emerge_progress


Please improve it in any way that you see fit, and remove this notice {{Cleanup}} from the article. For tips on cleaning and formatting see Cleanup process



Please format this article according to the guidelines and Wikification suggestions, then remove this notice {{Wikify}} from the article


Contents

Introduction

Get an approximate status of a current running emerge job to see how far along packages have compiled.

The Make Way

This approach works by running make twice, once in a pretend run to figure out what files the package is going to try to compile and then again for real. In THEORY it should be more accurate then trying to guess what files will need compiled and faster. However some packages don't use make and a pretend run may not give a full picture of everything make will actually do. For these reasons estimating by file-counting should work better since it works for all kinds of compiled packages and new file extensions can be added easily.

Warning: The script may exit without displaying anything or may display percent values greather then 100%. If your experiencing this problem, try the solution provided by TheJackal: Minor BugFix 1.


# cp /usr/lib/portage/bin/emake /root/emake.backup
Warning: Don't use this on a multi-user or untrusted system. There is a serious security issue at this script, see talk page for details
File: /usr/lib/portage/bin/emake
#!/bin/bash
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: emake 3483 2006-06-10 21:40:40Z genone $
#
# emake: Supplies some default parameters to GNU make. At the moment the
#        only parameter supplied is -jN, where N is a number of
#        parallel processes that should be ideal for the running host
#        (e.g. on a single-CPU machine, N=2).  The MAKEOPTS variable
#        is set in /etc/make.globals.  We don't source
#        /etc/make.globals here because emake is only called from an
#        ebuild.
#
# make ${MAKEOPTS} ${EXTRA_EMAKE} "$@"

#package name
name=$(basename $(pwd))

if [ -f /tmp/emerge.$name.build ]; then
  make ${MAKEOPTS} ${EXTRA_EMAKE} "$@"
else
  make -n -i -w ${MAKEOPTS} ${EXTRA_EMAKE} "$@" 2>&1 | egrep "^g?make\[?" > /tmp/emerge.$name.build
  make -w ${MAKEOPTS} ${EXTRA_EMAKE} "$@" | tee /tmp/emake.$name.edone
fi


What this does is redirect the output of a pretend make (note the -n) with the current dirs (note the -w) of each step to a temporary file and save the output of the real make run to both a file and to the standard output. There is nothing special in the above code; the real magic is done next.


File: ~/eprogress
#!/bin/sh
#Created by michael brown, Emerge Progress revision 2.
#Created with help from xmb(API'ed MicrosuxX Bot - http://xmb.ath.cx) from #bash

#If you don't know what it does, don't mess with it :) -- really, I mean it. ... If you
#have any problems with the code, feel free to email me '''(xxxmikey_bxxx{at}hotmail{dot}com)'''
#with a good subject like "Emerge-progress."  : )

#functions:

function makeprogress {
  #name="`basename $entry .etodo`";
  name=$1;
  A="/tmp/emake.$name.etodo"; #todo
  B="/tmp/emake.$name.edone"; #Done so far
  todo=""
  done=""

  #now for the clever bit ;)
  if [ -e /tmp/emake.$name.edone ]
  then
    todo=$(wc -l $A)
    done=$(gawk -v m="$(tail -n 1 $B)" 'BEGIN { m = gensub(/([][/])/, "\\\\\\1", "g", m) } $0 ~ m { print FNR }' $A)
    done="${done##* }"
  fi

  if [ "$done" != "" ]
  then
    percent=$((100 * $done / $todo))
  else
    percent="-"
    done="-"
  fi
 
  if [ "$todo" == "" ]
  then
    todo="-"
  fi


  name=${name:6}
  len=${#name}; len=$((30-len)); i=0; sp=""
  for (( i = 0; i <=len; i++ )); do sp="$sp "; done

  echo "$sp $done/$todo, ($percent%)"
}

function decodetree {
    action="?"
    pid=""
    title=$1
    input=$(echo "$1-")

    function=$(echo $input | grep -i -e "sandbox(")
    if [ ${#function} -gt 0 ]
    then
      pid=${input##*sandbox\(};
      pid=${pid%%\)*};
      title=$(ps w --no-headers --format "%a" $pid)
      title=${title##*\[};
      title=${title%%\]*};
    fi

    #compiling
    function=$(echo $input | grep -i -e "make(")
    if [ ${#function} -gt 0 ]
    then
      action="M"
      title=$(echo "$title $(makeprogress $title)")
    fi

    #compiling
    function=$(echo $input | grep -i -e "rsync(")
    if [ ${#function} -gt 0 ]
    then
      action="$"
    fi
 
    #Downloading...
    function=$(echo $input | grep -i -e "-wget(")
    if [ ${#function} -gt 0 ]
    then
      pid=${input##*wget\(};
      pid=${pid%%\)*};
      title=$(ps w --no-headers --format "%a" $pid)
      title=$(basename "$title")
      action="D"
    fi

    #Configuring...
    function=$(echo $input | grep -i -e "-configure(")
    if [ ${#function} -gt 0 ]
    then
      action="C"
    fi

    function=$(echo $input | grep -i -e "-prepstrip(")
    if [ ${#function} -gt 0 ]
    then
      action="S"
    fi
 
    #final config
    function=$(echo $input | grep -i -e "-ldconfig(")
    if [ ${#function} -gt 0 ]
    then
      action="F"
    fi

    function=$(echo $input | grep -i -e "-ld(")
    if [ ${#function} -gt 0 ]
    then
      action="L"
    fi
 
    function=$(echo $input | grep -i -e "-portageq(")
    if [ ${#function} -gt 0 ]
    then
      action="P"
    fi

    echo "[$action] $title"
}

# Notes:
# Well, this baby does everything it should, and it's impossible to go above 100%.
# If it does, PLEASE PLEASE tell me, : ) as it's a serious problem!

res=$(pgrep -x emerge | grep -v $$)

echo "Emerge processes: (for debugging)"
for entry in $res
do
  etree=$(pstree -A -l -p $entry)
  eres="$(decodetree "$etree")"
  if [ "$eres" == "" ]
  then
    echo "$etree"
  else
    echo " * $eres"
  fi
done

exit 0;


# mv eprogress /usr/local/bin/ && chmod +x /usr/local/bin/eprogress

When you run it, you will soon discover many symbols:

[M] = Compiling the build ...
[$] = It's a portage sync : )
[D] = wget is downloading it for you : )
[C] = Project is being configured
[L] = Linking objects
[S] = Strip command (I don't know what it does! I just found it!)
[F] = This is when emerge configures your enviroment to handle the new builds. F for finished : )
[P] = This is a portage query
[?] = Means it's unknown
NA: [-] = Means it's doing nothing or the pid is not found (one implies the other)

Example output:

Emerge processes: (for debugging)
[D] |-xterm(16781)---bash(16796)---su(20442)---bash(20553)---emerge(19264)---wget(22266)

Emerge Progress (v2): (old)
 * (V)[D] (22266) thunderbird-1.0.2-source.tar.bz2
 * (L)[-] eject-2.0.13:                     31/31, (100%)
 * (L)[-] mozilla-thunderbird-1.0.2:        -/616, (-%)
 * (L)[-] xmule-1.8.4-r1:                   291/291, (100%)

This output is from when thunderbird-1.0.2-source was downloading.

Nota bene: The funky stuff at the top is for other people helping with this project. It's a tree of the processes being run by emerge. The numbers are the pids (To get information about a pid ... ps w 16796) Notice mozilla-thunderbird-1.0.2: -/616, (-%). This means that the last entry on the .edone could not be found. It has something to do with the pretend make. Since there are over 6,000 source-code files to compile, if someone could enlighten me as to why this happens, please email me! The log file has not yet been removed because this code is still quite testy.

Minor BugFix 1

For my revision of portage, the gawk line above simply spits a series of progressive numbers to the console, and so does not work.

I solved the problem into a "works-good-enough" stage by replacing the lines

done=$(gawk -v m="$(tail -n 1 $B)" 'BEGIN { m = gensub(/([][/])/, "\\\\\\1", "g", m) } $0 ~ m { print FNR }' $A) done="${done##* }"

with

"done=$(wc -l $B)."

Unfortunately this fix only makes it work. Because two big files are read, it produces a rather big slowdown.

I hope this is helpful for all others who encounter the same bug.

--TheJackal 11:27, 20 October 2005 (GMT)

Got the same problem, thanks for fixing! - Max, 23 Oct 05

Bug in the BugFix this bug's not quite so bad as the one it fixes, so it's still a bugfix, but it suffers the same bug as all the C++ / C versions as it now produces more than 100% for some merges :S, so I'm currently trying to figure out what's wrong with gawk.

--TheJackal 20:58, 23 October 2005 (GMT)

The File count Way

This approach works by counting files as such it depends on guessing what files are probably going to need to be compiled and what files have probably already been compiled. This is achieved by looking at the file extension of files. New file extensions can be added so its a little more flexiable then depending on a makefile.

I'm kind of amazed nobody mentioned this here before, but the file count way is theoretically flawed. There is no way to make a trustworthy prediction on the amount of output files from the amount of source files. The chance your build will end at 80%, 275%, or some other arbitrary number is pretty likely. The scripts presented here for you are all a display of horrid scripting habits, mainly to do with quoting issues. Can somebody *please* fix this page up?

Simple Version

File: /usr/local/bin/eprogress
#!/bin/sh
source /etc/make.globals
source /etc/make.conf

CUR=`tail -n 5 /var/log/emerge.log | grep Compiling | sed 's/\(.*\) Compiling\/\(Merging\|Packaging\) (.*\/\(.*\)::.*)/\3/'`
TOT=`find $PORTAGE_TMPDIR/portage/$CUR -iname "*.c*" | wc -l`
PRG=`find $PORTAGE_TMPDIR/portage/$CUR -iname "*.o*" | wc -l`
PER=`echo $PRG $TOT | gawk '{ sum += ($1 / $2) * 100 }; END { print sum }' -`
echo "Currently Compiled Package: " $CUR " (" $PER "%)"
# chmod +x /usr/local/bin/eprogress

After that you can watch progress in %, by executing the following command:

# watch -n 5 --no-title eprogress

-n 5 sets refresh interval to 5 seconds and --no-title removes title.

If you have osd_cat (emerge xosd), you can do this:

# eprogress | osd_cat --colour=white --outline=1 --outlinecolour=black

Set that to a hotkey in your WM, and you have an OSD emerge progress hotkey :)

Newer versions of osd_cat have support for percentages. You can have it draw you a bar graph of completion instead. This next command will draw a bar of completion for the first package returned by eprogress (usually the one merging):

eprogress | grep ^\ \ \* | head -n 1 | sed 's/.*\] //' | sed 's/:[^(]*(/ /' | sed 's/%)$//' | \
awk '{system("osd_cat --colour=white --outline=1 --outlinecolour=black -b percentage -P "$2" -T "$1);}

Improved version!

As with the previous code, you can use watch, but watch doesnt support colours, so you must use the -nc option

# watch -n 5 eprogress -nc
File: /usr/local/bin/eprogress
#!/bin/sh
#By Michael Brown
#
#I've been working on make improvements on displaying the current emerge status,
#this is the code I've come up with so far. It's still in testing, but it works
#when it compiles java as well.. soon adding perl, py etc..
#
#Any problems or fixes please email me! xxxmikey_bxxx{AT}hotmail{dot}com
#
#If it gets any good i may rewrite it into C.

#get PORTAGE_TMPDIR <- hard coded portage location?, might want to change!
source /etc/make.globals
source /etc/make.conf > /dev/null 2>&1

notitle=0; nocol=0;

for i in $@; do
  if [ "$i" == "-nt" ]; then notitle=1; fi
  if [ "$i" == "-nc" ]; then nocol=1; fi
  if [ "$i" == "--help" ]; then
    echo -e "Emerge Progress Script V1:\nThis code was a collection of other code
 with extras,\nrecoded by Michael Brown, Previous authors plasmagunman, kopfarzt
n\nDesigned to display the emerge status\nUsage: eprogress [options]\n\nOp
ons:\n  -nc Enables the no colour mode\n  -nt Enables the no title mode\n  --help
displays this text.\n\n\nDisplay:\nStandard format of result:\n  * [1] Packagena
   todo/outof (percent%).\n\n[1] will either show a [C] for compiling, or [x] fo
cancelled.\nPackage is the package you are emerging\ntodo/outof is the number of
iles for that emerge left todo, and the total number."
    exit 0
  fi
done

if [ "$notitle" -eq "0" ]; then echo -e "Emerge Progress:"; fi

if [ "$nocol" -eq "0" ]; then
  coltab1="\E[;32m"; coltab2="\033[1m * \033[0m"
else
  coltab1=""; coltab2="  *"
fi

#can i read tmp folder?
folder=`find $PORTAGE_TMPDIR/portage/ -name work 2> /dev/null`

if [ $? -gt 0 ]; then
  if [ "$nocol" -eq "0" ]; then echo -en '\E[;31m'; fi
  echo "!! You do not have access to the portage tmp folder, unable to load stat !!"
  tput sgr0
fi

#set up extentions
srcext="c cpp cc java C"
objext="o class"

for ext in $srcext; do srccmd="$srccmd -name *.$ext* -o"; done
for ext in $objext; do objcmd="$objcmd -name *.$ext* -o"; done

for WORKDIR in $folder; do
  TODO=`find $WORKDIR $srccmd -false 2> /dev/null | wc -l`
  DONE=`find $WORKDIR $objcmd -false 2> /dev/null | wc -l`

  if [ "$((TODO))" == "0" ]; then
    percent=0
  else
    percent=$((100 * $DONE / $TODO))
  fi

  name="`basename \`dirname $WORKDIR\``"
  len=${#name}; len=$((30-len)); i=0; sp=""
  for (( i = 0; i <=len; i++ )); do sp="$sp "; done

  #is it currently compiling?
  res=$(ps ax | grep -v grep | grep -i $name)
 
  if [ ${#res} -gt 0 ]; then
    action="C"
  else
    action="x"
  fi

  echo -e $coltab1 "$coltab2 [$action] $name: $sp $DONE/$TODO, ($percent%)"
done

if [ "$nocol" -eq "0" ]; then tput sgr0; fi
exit 0

The above shell code is bugged in so many ways it's just not funny anymore. Somebody please fix its syntax up or remove it so nobody needs to see it anymore.

C++ version

This version relies on the /proc filesystem for better speed, but currently doesn't do colors. If you don't have the /proc filesystem then this version won't work. Later Ppid identifiers may be used instead.

# g++ -o eprogress eprogress.cpp && cp eprogress /usr/local/bin/
Code: eprogress.cpp
// Updated by David K. (David-Dot-Kelley-At-comcast-dot-net)
// Updated by Annoymance, phox and darklama

#include <sys/types.h>
#include <dirent.h>
#include <unistd.h>
#include <fstream>
#include <string>
#include <cstdio>

using namespace std;

#define srccount (sizeof(src)/sizeof(src[0]))
#define objcount (sizeof(obj)/sizeof(obj[0]))

const char *src[] = {".c",".cc",".cpp",".c++",".C",".java",".s",".l",".y"};
const char *obj[] = {".a",".o",".class",".so",".ko",".lib"};

class path {
  DIR *d,*t;
  struct dirent *e;
  
  public:
    path(char *p) {d=opendir(p); t=opendir("."); if (d) chdir(p); e=NULL;}
    ~path() {closedir(d); if (t) fchdir(dirfd(t)); closedir(t);}
    void go() {chdir(e->d_name);}
    void up() {chdir("..");}
    bool isdir() {return e->d_type==DT_DIR;}
    operator char*() {return e ? e->d_name : 0;}
    bool operator++(int a) {
      while (d && (e=readdir(d))) {
        if ((a=strspn(e->d_name, "."))<=2 && e->d_name[a]=='\0')
          continue;
        return true;
      }
      return false;
    }
};

void countfiles(char*p,unsigned&todo,unsigned&done) {
  path d(p);
  char *s;
  
  while (d++)
    if (d.isdir()) {
      countfiles(d,todo,done);
    } else {
      for (unsigned c=0; c<srccount; c++)
        if ((s=strstr(d, src[c])) && *(s+strlen(src[c])) == '\0') todo++;
      for (unsigned c=0; c<objcount; c++)
        if ((s=strstr(d, obj[c])) && *(s+strlen(obj[c])) == '\0') done++;
    }
}

int findproc(char *name) {
  // Find the proc of portage :)
  char buffer[256];
  path p("/proc");
  
  while (p++)
    if (p.isdir()) {
      p.go();
      ifstream rfile("cmdline");
      if (rfile.is_open()) {
        rfile.getline(buffer,sizeof buffer);
        if (strstr(buffer,name) != NULL)
          return 0;
      }
      p.up();
    }
  return 1;
}

int main() {
  unsigned int done,todo,perc;
  path p("/usr/tmp/portage/");
  
  puts("Emerge Progress v4.21 C++:");
  
  while (p++)
    if (p.isdir()) {
      p.go();
      todo=done=0; countfiles("/work",todo,done);
      if (todo>0 && done<=todo) {
        perc=(done/todo)*100;
        printf("[%c] %s: %d/%d (%d%%)\n","XC"[findproc(p)],static_cast<char*>(p),done,todo,perc);
      }
      p.up();
    }
  return 0;
}

Minor Bugs in the c++ program

t35t0r

The c++ program works for the most part but sometimes I get errors like this:

 Emerge Progress v2. C++ version:
 * [C] tunepimp-0.3.0: 80/49, (163%)

..and on another gentoo box I get items that aren't being emerged beginning with an [X] with completon 0% or partially completed (<100%). Now those can't be items which I've prematurely ctrl+c out of, because it doesn't occur on some boxes (like the one where I get the >100%).

Cause this program just counts object files for percentages and it's possible to have more object files than c files.

Similar, but different functionality is got from using genlop -c (emerge genlop).

checkprogress.sh (c) Sadysta

Code: checkprogress.sh
#!/bin/sh
# Copyright 2006 Sadysta
# Distributed under the terms of the GNU General Public License v2

if [ ! -x /usr/bin/time ]; then
  echo -e "\33[1;31m!!\33[m You must have sys-process/time installed."
  exit 1
fi

OLD_FULL="-"
FULL=$1
while true; do
  if [[ "$1" == "" ]]; then
    FULL=`tail -n 2 /var/log/emerge.log | grep Compiling | sed "s/\(.*\)::\(.*\)/\1/" | sed "s/.*(//"`
  fi
  if [[ "${FULL}" != "" ]]; then
    if [[ "${FULL}" != "${OLD_FULL}" ]]; then
      if [[ "$1" != "" ]]; then
        echo -e "\33[1;33m!!\33[m Manual build directory override. Entering scanning only mode..."
      fi
      echo -e "\33[1;32m**\33[m Compiling ${FULL}..."  
      OLD_FULL=${FULL}
    else
      echo -e "\33M\33[1A"
    fi
    if [ -d /var/tmp/portage/${FULL} ]; then
      SRC_COUNT=`/usr/bin/time -p -o /tmp/chkprgrs_t1.txt find /var/tmp/portage/${FULL} -regextype posix-extended -regex ".*(\.cpp|\.c|\.cc|\.asm|\.s|\.cxx|\.java|\.l|\.y)" | wc -l`
      READY_COUNT=`/usr/bin/time -p -o /tmp/chkprgrs_t2.txt find /var/tmp/portage/${FULL} -regextype posix-extended -regex ".*(\.o|\.a|\.class|\.ko|\.lib|\.so)" | wc -l`
      TIME1=$((`cat /tmp/chkprgrs_t1.txt | grep real | cut -d" " -f2 | sed "s/\.//" | sed "s/^0*\(.\)/\\1/"` / 10))
      TIME2=$((`cat /tmp/chkprgrs_t2.txt | grep real | cut -d" " -f2 | sed "s/\.//" | sed "s/^0*\(.\)/\\1/"` / 10))
      TIME=$((${TIME1} + ${TIME2}))
      if [[ ${TIME} -le 5 ]]; then
        TIME=5;
      fi
      if [[ ${SRC_COUNT} -eq 0 ]]; then
        SRC_COUNT=1;
      fi
      PERCENT=$((${READY_COUNT}*100/${SRC_COUNT}))
      echo -e "\33[1;32m**\33[m Done: ${READY_COUNT} of ${SRC_COUNT} (${PERCENT} %)..."
      sleep ${TIME}
    else
      echo -e "\33[1;33m!!\33[m Sources haven't been unpacked yet."
      sleep 5
    fi
  else
    if [[ "${FULL}" != "${OLD_FULL}" ]]; then
      echo -e "\33[1;33m**\33[m No compilation is in progress at the moment."
      OLD_FULL=${FULL}
    fi
    sleep 5
  fi
done

External Links

From Gentoo Forums:

Retrieved from "http://www.gentoo-wiki.info/Emerge/Watch_emerge_progress"

Last modified: Thu, 21 Aug 2008 03:55:00 +0000 Hits: 42,939