Re: [PLUG] Odd script problem

On Sun, Apr 3, 2016 at 11:22 AM, Fred Stluka <fred@bristle.com> wrote:

Matt,

If you set up a look like that, be sure to add a small delay like:
sleep 1

Also, add an echo statement, with perhaps a counter of how
many iterations you've tried, and perhaps a test to give up
after some number.

Otherwise, you're creating a tight CPU loop run from a cron
job, which may bring your system to its knees with no hint
of why.

And, yeah, the problem is almost certainly going to be
something about your environment, so calling printenv from
the cron job and comparing it with the printenv from an
interactive shell may be very enlightening.

--Fred
------------------------------------------------------------------------
Fred Stluka -- mailto:fred@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.
------------------------------------------------------------------------
On 4/3/16 11:00 AM, Matt Murphy wrote:

Followup:
"set -x" was excellent advice! It helped me realize I was actually getting back 255 from my failed PHP runs (which means my cli php.ini has problems as my log was bare. ) bash accepts errors only to 254, so I expect I'll find php was saying '500', but I'm not there yet. I do have what appears to be a solution to my problem. brute force failure detection:

php -c /etc/php5/cli/php.ini -d debug_errors -f index.php
while [ $? -gt 0 ]
do
php -c /etc/php5/cli/php.ini -d debug_errors -f index.php
done
exit $?

On Mon, Mar 28, 2016 at 5:27 PM, Morgan Jones <morgan@morganjones.org <mailto:morgan@morganjones.org>> wrote:

I would discourage sourcing anything in ~ for scripts that are
intended to be deterministic. Months from now you may make a
change to your environment which will break your script. It’s
also a security risk as you’re allowing an attacker access to a
script that might be running as root by editing a presumably less
privileged user’s environment.

I would suggest you define the environment for the script
deterministically whether it be in the script it self or in a
profile that is not tied to an interactive user.

-morgan

On Mar 28, 2016, at 16:31, Eric Lucas <eric@lucii.org
<mailto:eric@lucii.org>> wrote:

A common problem with cron scripts is that they are started with
/bin/sh and a nearly blank environment.
I usually source ~/.bash_profile or ~/.bashrc to set up PATH at
the least.
Alternatively, you can specify the full path of any command:
/usr/bin/php -c /etc/php5/cli/php.ini -d debug_errors -f script.php

If it works sometimes and fails sometimes that's a nasty problem
probably unrelated to the environment
issue but setting it up can't hurt.

Eric

On Mon, Mar 28, 2016 at 2:51 PM, Matt Murphy <mattyhead@gmail.com
<mailto:mattyhead@gmail.com>> wrote:

Thanks, Michael. That sounds pretty frustrating :)

Here, what happens is cron calls a bash script each minute
which simply does :

---------------------------------
#!/bin/bash

cd /path/to/script
date >> log
php script.php
---------------------------------

Actually, I that call is currently something more explicit like:
php -c /etc/php5/cli/php.ini -d debug_errors -f script.php

And sometimes php doesn't execute at all.

MM

On Mon, Mar 28, 2016 at 12:14 PM, Michael Lazin
<microlaser@gmail.com <mailto:microlaser@gmail.com>> wrote:

How are you running the php scripts? I work at a hosting
company and a common problem I see when users run php
scripts with cron is they use something like curl which
which resolves at the dns level and makes a request back
to the same webspace artificially inflating web traffic
and creating unecessary server load. I recommend if you
want to run a local php script with cron simply run it
locally with the php command. It makes way less overhead.

On Mon, Mar 28, 2016 at 11:18 AM, Matt Murphy
<mattyhead@gmail.com <mailto:mattyhead@gmail.com>> wrote:

Greetings!

I'm a long-time linux user and frequent defacto admin.

I have a problem: a cron job is running that calls a
bash script to change the directory and run a php
script which, in turn, scrapes an old-and-ugly site
which our office owns but does not control (meaning
that doing something sensible like serving the data
we now scrape via an API is not an option). With the
size of the content currently being scraped, the run
time is one or 2 seconds. The cron job runs, at
peak, once a minute.

Naturally, I do some logging: The bash script writes
out a timestamp to log when it runs, the PHP script
writes 'locked' to a status file and 'working' and a
timestamp to log. Any exceptions are captured and
logged out with timestamp when they occur. End-of
run is also logged, with timestamp and, finally,
'unlocked' is written to the status file.

The problem: I'm showing some instances when the
cron job runs and the php job doesn't, at all (no
status written, no 'working' log entry). It's not
being caused by collision (recall the scrape runs in
under 2 seconds and the cron interval is 1m and
there's app-level logging-o-plenty to expose a
collision); additionally, I've seen similar at longer
increments (2, 5 mins). PHP error level is set to
E_ALL and there is no output that corresponds to the
missed runs.

My thinking is that I might be looking at a failure
of one scripting environment to hand off to another.
Has anyone seen something like this? I'm having
trouble finding system log activity that explains
what I'm seeing.

Matt Murphy
Philadelphia Elections Commission

___________________________________________________________________________
Philadelphia Linux Users Group --
http://www.phillylinux.org <http://www.phillylinux.org/>
Announcements -
http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion --
http://lists.phillylinux.org/mailman/listinfo/plug

-- Michael Lazin

to gar auto estin noein te kai ennai

___________________________________________________________________________
Philadelphia Linux Users Group --
http://www.phillylinux.org <http://www.phillylinux.org/>
Announcements -
http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion --
http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
<http://www.phillylinux.org/>
Announcements -
http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion --
http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements -
http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion --
http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements -
http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion --
http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug