JP Vossen on 22 Nov 2011 22:10:53 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Finding a /etc/group entry containing a user list

Sorry I took so long on this thread, it was just one of those weeks... Or two...
This is quite long, since I'm replying to a bunch of answers in one place, though I've trimmed a lot too. But to cut to the chase, and for those who took one look at these "one-liners" and started grumbling about line-noise or cats walking on keyboards (or dogs in Jeff's case :)...
Often in the "shell" (or really, the gazillion little Unix and/or GNU tools that come with the system), There Is More Than One Way To Do It. (TIMTOWTDI is also a Perl truism.) 

So let's take a few of these apart and see how they work...
Both of the following assume the list of users to match is in the $users variable, while I originally had a file. No matter, we can do it either way. These then spit out a list of each group each user belongs to and "sum them up." 

### Sean's
for group in `groups $users | cut -d: -f2-`; do echo $group; done | sort | uniq -c | sort -nr 

### Mark's
groups $users | cut -d: -f2-|tr " " "\012" |sort | uniq -c | sort -nr


If I just hard code some users for an example, it looks like this:
$ groups jp user1 | cut -d: -f2-|tr " " "\012" |sort | uniq -c | sort -nr
      2 data
      2
      1 video
      1 sudo
[...]


So for Sean's
for group in `groups $users | cut -d: -f2-`; do echo $group; done | sort | uniq -c | sort -nr 

1) Command substitution: `groups $users | cut -d: -f2-`
Lists the groups for each user (groups $users), then cuts off the user name at the ':' delimiter, printing only the "second" field (cut -d: -f2-). `` is the old way, $() is the new way. I like the new way because I find it slightly easier to read and nest, but it doesn't really matter and the old way is more portable.
Command substitution (http://en.wikipedia.org/wiki/Command_substitution) is a really key concept on shell (& other) scripting and is a way to get the output from one command or series of commands (i.e. a "pipeline") into the middle of another command. Once you get the hang of it, it's amazingly useful.
2) The for...do...done part is a shell for loop that just prints each group name on a line by itself, otherwise you get: 
	jp : jp adm dialout cdrom floppy sudo audio video plugdev data
3) The '| sort | uniq -c | sort -nr' part pre-sorts the line-by-line lists, then 'uniq -c' counts the duplicate (common!) groups, then the final 'sort' sorts the group counts to give the highest first.
Mark's does exactly the same thing, except for without the command substitution and for echo loop. Instead, he just brute-force translates (tr) spaces into newlines. 


Now for the one we ended up with at the very bottom:
for user in $(cat userlist); do id -Gn $user; done | tr " " "\012" | sort | uniq -c | sort -rn | egrep -v '^[[:space:]]+1 '
1) $(cat userlist) is the new command substitution, and it inserts the contents of the 'userlist' file into the middle of the for loop. So we loop over each line (user) in that file, assigning each to $user in turn.
2) 'id -Gn $user' just runs the 'id' command for each user, and returns the name (n) of all the groups (G) for that user. That's in case the 'groups' command used above doesn't allow more than 1 argument (as in Gentoo it sounds like). 

3) We then brute-force space into newline using '| tr " " "\012" '

4) Feed that list of groups to the '| sort | uniq -c | sort -nr' as above.
5) And finally remove any groups that have only 1 hit, just to make the list shorter: | egrep -v '^[[:space:]]+1 ' This is an egrep regular expression, here '^' is the start-of-line anchor, [[:space:]] is the character class for space or tab, '+' means one or more, '1' is a literal '1' and the '-v' is the inVerse. IOW, remove any line that starts with one or more white space characters then a 1. 

I hope this breakout is useful and doesn't just bore the list to tears...


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thread details:


Date: Sat, 12 Nov 2011 16:01:49 -0500
From: Rich Freeman<r-plug@thefreemanclan.net>

How about this (typing on phone so can't debug much). Put list of users in
a file, one per line. Feed one line of group at a time into grep -o -f
userlist -c (or pipe into wc -l if the -c isn't right) to get a per-line
count. Probably a little xargs and a function or script to split the
groups, and maybe a little awk to print the group names.
I was going to argue that the '-o' part will return *only* the username, but then I re-read the "feed /etc/group" one line at a time" part. Hmmmmm. 

OK, odd.  This *should* work, but does not (bash 4.1.5 on Debian Squeeze):
while read line; do echo $line; grep -o -f userlist -c; done < /etc/group
If you remove the grep part, it works as expected (prints every line from /etc/group). But as soon as the grep is added, it does only the first line. I should probably know why that's happening, but at the moment I don't... :-) 



------------------------------
Date: Sat, 12 Nov 2011 18:41:29 -0500
From: Fred Stluka<fred@bristle.com>



Yeah, I'd think you usually want to create a new group unless
there's an existing group that really does serve the same purpose.
Otherwise, you end up with one group serving 2 purposes, which
may diverge in the future, and then you'd have a mess.  Also, as
you said, it will often matter that extra people are in the group
when they don't need to be.  So, I think the only time you'll
really use this script will be to find out whether there is
already a group for this purpose.


Yup, I did end up having to create a new group.



How often do you have to do this, that chaining greps as an ad hoc
on-the-fly solution isn't good enough?  Sounds like you've inherited
a system with a ton of groups with non-obvious names.  Bummer!
This was the first time this particular question occurred to me. And yes, the groups are a bit...ugly.
I started to say it wasn't nearly as good a question as I thought it was at first. That's true from a purely practical standpoint, but this thread demonstrates a whole bunch of good things about the Linux, Unix and open source worlds, and from that perspective it's well worth it. 



------------------------------
Date: Sat, 12 Nov 2011 22:57:28 -0500
From: bergman@merctech.com

In the message dated: Sat, 12 Nov 2011 18:41:29 EST,
The pithy ruminations from Fred Stluka on
<Re: [PLUG] Finding a /etc/group entry containing a user list>  were:
=>  >  Of course, the flip side is that other users in group15 will have
=>  >  perms they don't need.  I don't really care too much for this
=>  >  purpose, but usually that would matter.  So maybe this question is
=>  >  less useful than I thought at first and yet-another-group really
=>  >  is the best answer.

Not always. Don't forget that users were limited to being in 16 groups, and
many programs, paricularly NFS, use structure with that limit.[1]. The
various tools (usermod, getent, etc.) will allow you to 'add' a user to more
groups and will report that the user is in those groups...but things that
depend on that group membership won't always work.

	[1] http://blogs.oracle.com/peteh/date/20050614
	(get past the icky 'oracle.com' URL that used to read
	'sun.com', and realize that much of that--notably the NFS stuff--does
	apply to Linux)
Not in this case since it was all local, but that's still a really good point. 



I wouldn't 'parse /etc/group' directly, as group info could be stored in
other places (NIS tables, LDAP, etc). It's safer to use "getent" (which
will use the data sources&  precedence defined in /etc/nsswitch.conf)
to retrieve group information.
Again, in my quick&dirty case it doesn't matter, but still an excellent point. Hummm, except... On my Lenny system 'getent' only gives me the primary group, which is not what I need. 



You could probably do something vaguely like this pseudo-code:

<snip>
I did something kinda like that in Perl, where my high-scoring group was 6 out of 10 and when I looked closely, there were lots of other folks in that group that didn't need access, so I ended up using a new group anyway. Sigh. 




------------------------------
Date: Sun, 13 Nov 2011 09:24:55 -0500
From: bergman@merctech.com
Subject: Re: [PLUG] Finding a /etc/group entry containing a user list
To: "Philadelphia Linux User's Group Discussion List"
	<plug@lists.phillylinux.org>
Message-ID:<13645.1321194295@localhost>
Content-Type: text/plain; charset="us-ascii"

In the message dated: Sun, 13 Nov 2011 14:10:49 +0100,
The pithy ruminations from sean finney on

=>  something like this?
=>

### Sean's

=>  for group in `groups $users | cut -d: -f2-`; do
=>  	echo $group
=>  done | sort | uniq -c | sort -n
Yup, THAT's what I was trying to think of!!! I knew there had to be a way, but... Cool! 



Nice... or slightly more compact:


### Mark's

   groups $users | cut -d: -f2-|tr " " "\012" |sort | uniq -c | sort -nr


Ditto.  I can't decide which of the above is more readable.




------------------------------
Date: Sun, 13 Nov 2011 09:40:22 -0500
From: Rich Freeman<r-plug@thefreemanclan.net>

On Sun, Nov 13, 2011 at 9:24 AM,<bergman@merctech.com>  wrote:

Nice... or slightly more compact:

? ? ? ?groups $users | cut -d: -f2-|tr " " "\012" |sort | uniq -c | sort -nr


Apparently not all groups implementations support multiple arguments
(like the one standard on Gentoo :) ).  So, that might need to be
tweaked on some distros.  From the very little that I've been able to
google I'm not sure that this command is even all that standardized.


Bummer!  Works on Lenny & CentOS-5 (target env.).



Date: Sun, 13 Nov 2011 22:52:31 +0100
From: sean finney <seanius@seanius.net>

well if we're talking about being portable we probalby shouldn't be
using shell snippets but instead python/perl/C.  I don't think there's


LOL.  Very true!  I *hate* trying to write "portable" shell scripts!



any POSIX defined method for this (even id -Gn, which would be the
equivalent of groups, is not defined in POSIX afaik, though may be more
reliable between linuces).  Even then it only supports only one user at
a time, but that's not really such a hinderance, just a little more foo
in the shell snippet:

	for g in $(for u in $(echo $users); do id -Gn $u; done); do echo $g; done | sort | uniq -c | sort -n

arguably this is starting to stretch the boundaries of what could be called
a "one-liner"
Wow, now that's an ugly one! Which is another way of saying, Very Nice! :-) You can reduce that one like this, which might be slightly easier to read:
for user in $users; do id -Gn $user; done | tr " " "\012" | sort | uniq -c | sort -rn 

Actually, I'd probably do (tested Lenny & CentOS-5):
for user in $(cat userlist); do id -Gn $user; done | tr " " "\012" | sort | uniq -c | sort -rn | egrep -v '^[[:space:]]+1 '
Thanks to everyone who thought about this & replied. I've made a note of this thread and it could very well end up in the 2nd edition of the cookbook (suitable cited), if/when. (Though you start getting into questions about how much of this is "bash" vs. GNU Coreutils or GNU C, etc.) 

Later,
JP
----------------------------|:::======|-------------------------------
JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
My Account, My Opinions     |=========|      http://www.jpsdomain.org/
----------------------------|=========|-------------------------------
"Microsoft Tax" = the additional hardware & yearly fees for the add-on
software required to protect Windows from its own poorly designed and
implemented self, while the overhead incidentally flattens Moore's Law.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug