Matt Murphy on 3 Apr 2016 12:04:24 -0700


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Odd script problem


Eep, I forgot Sony was out there poisoning the PS/d search space.  Ok, I have some rich references now...

Thanks, JP.  I'll look at PS4!

MM

On Sun, Apr 3, 2016 at 11:32 AM, JP Vossen <jp@jpsdomain.org> wrote:
`set -x` is great.  You can also do `set +x` to turn it back off.  Or you can do `bash -x buggy_script` to debug without actually editing the script.  You can do the same for `set -v` (verbose) but I've always found -x much more useful.

PS4 is the debug prompt.  The default isn't that great, I like one of these, though some may argue they are overkill and/or to noisy:
        PS4='+xtrace $LINENO: '
        PS4='+xtrace $BASH_SOURCE::$FUNCNAME-$LINENO: '

NOTE the first character is shown as many times as necessary to denote the shell nesting depth, so make it something obvious like "+' or '*' or '&'.



On 04/03/2016 11:00 AM, Matt Murphy wrote:
Followup:
"set -x" was excellent advice!  It helped me realize I was actually
getting back 255 from my failed PHP runs (which means my cli php.ini has
problems as my log was bare. )  bash accepts errors only to 254, so I
expect I'll find php was saying '500', but I'm not there yet.  I do have
what appears to be a solution to my problem.  brute force failure detection:

php -c /etc/php5/cli/php.ini -d debug_errors -f index.php
while [ $? -gt 0 ]
do
   php -c /etc/php5/cli/php.ini -d debug_errors -f index.php
done
exit $?



On Mon, Mar 28, 2016 at 5:27 PM, Morgan Jones <morgan@morganjones.org
<mailto:morgan@morganjones.org>> wrote:

    I would discourage sourcing anything in ~ for scripts that are
    intended to be deterministic.  Months from now you may make a change
    to your environment which will break your script.  It’s also a
    security risk as you’re allowing an attacker access to a script that
    might be running as root by editing a presumably less privileged
    user’s environment.

    I would suggest you define the environment for the script
    deterministically whether it be in the script it self or in a
    profile that is not tied to an interactive user.

    -morgan




    On Mar 28, 2016, at 16:31, Eric Lucas <eric@lucii.org
    <mailto:eric@lucii.org>> wrote:

    A common problem with cron scripts is that they are started with
    /bin/sh and a nearly blank environment.
    I usually source ~/.bash_profile or ~/.bashrc to set up PATH at
    the least.
    Alternatively, you can specify the full path of any command:
      /usr/bin/php -c /etc/php5/cli/php.ini -d debug_errors -f script.php

    If it works sometimes and fails sometimes that's a nasty problem
    probably unrelated to the environment
    issue but setting it up can't hurt.

    Eric

    On Mon, Mar 28, 2016 at 2:51 PM, Matt Murphy <mattyhead@gmail.com
    <mailto:mattyhead@gmail.com>> wrote:

        Thanks, Michael.  That sounds pretty frustrating :)

        Here, what happens is cron calls a bash script each minute
        which simply does :

        ---------------------------------
        #!/bin/bash

        cd /path/to/script
        date >> log
        php script.php
        ---------------------------------

        Actually, I that call is currently something more explicit like:
        php -c /etc/php5/cli/php.ini -d debug_errors -f script.php

        And sometimes php doesn't execute at all.

        MM

        On Mon, Mar 28, 2016 at 12:14 PM, Michael Lazin
        <microlaser@gmail.com <mailto:microlaser@gmail.com>> wrote:

            How are you running the php scripts?  I work at a hosting
            company and a common problem I see when users run php
            scripts with cron is they use something like curl which
            which resolves at the dns level and makes a request back
            to the same webspace artificially inflating web traffic
            and creating unecessary server load.  I recommend if you
            want to run a local php script with cron simply run it
            locally with the php command.  It makes way less overhead.

            On Mon, Mar 28, 2016 at 11:18 AM, Matt Murphy
            <mattyhead@gmail.com <mailto:mattyhead@gmail.com>> wrote:

                Greetings!

                I'm a long-time linux user and frequent defacto admin.

                I have a problem:  a cron job is running that calls a
                bash script to change the directory and run a php
                script which, in turn, scrapes an old-and-ugly site
                which our office owns but does not control (meaning
                that doing something sensible like serving the data we
                now scrape via an API is not an option). With the size
                of the content currently being scraped, the run time
                is one or 2 seconds.  The cron job runs, at peak, once
                a minute.

                Naturally, I do some logging:  The bash script writes
                out a timestamp to log when it runs, the PHP script
                writes 'locked' to a status file and 'working' and a
                timestamp to log.  Any exceptions are captured and
                logged out with timestamp when they occur.  End-of run
                is also logged, with timestamp and, finally,
                'unlocked' is written to the status file.

                The problem:  I'm showing some instances when the cron
                job runs and the php job doesn't, at all (no status
                written, no 'working' log entry).  It's not being
                caused by collision (recall the scrape runs in under 2
                seconds and the cron interval is 1m and there's
                app-level logging-o-plenty to expose a collision);
                additionally, I've seen similar at longer increments
                (2, 5 mins).  PHP error level is set to E_ALL and
                there is no output that corresponds to the missed runs.

                My thinking is that I might be looking at a failure of
                one scripting environment to hand off to another.
                Has anyone seen something like this?  I'm having
                trouble finding system log activity that explains what
                I'm seeing.

                Matt Murphy
                Philadelphia Elections Commission

            Michael Lazin
            to gar auto estin noein te kai ennai



Later,
JP
--  -------------------------------------------------------------------
JP Vossen, CISSP | http://www.jpsdomain.org/ | http://bashcookbook.com/

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug